One of the Go language’s selling points is its Goroutines. Goroutines are threads that are manufactured and managed in user-space. Goroutines are similar to Green Threads, but there is no need for a direct relationship between a given Goroutine and given kernel thread.

In order for Go to manage a potentially infinite number of Goroutines that could possibly exist at runtime it needs a way to organise which ready Goroutines to run and which Goroutines should wait, and this is the job of the scheduler.

The scheduler is quite simple and easy to understand. The key resources that it manages are:

• G - Goroutine.
• M - Worker thread, or machine (Kernel threads).
• P - Processor.

When the Go application starts it reads the GOMAXPROCS environment variable that determines the number of Ps that are able to be created by the runtime, with an exception that blocked Ps do not count toward the quota. Perhaps, then it would be better to think of the max value as the maximum number of unblocked Ps allowed to exist at any given point in time.

Each P has a runq, a queue of Goroutines that have been allocated to that P to run. When a Goroutine G is created, the scheduler looks for an empty P run queue (runq), and places the Goroutine into it, or, if all the runqs on all the Ps have work, the Goroutine is placed in a global runq, waiting for a P to run out of work.

When an P finds itself with no work to do it looks at the global runq, then the runqs of other Ps, this latter part is called job-stealing, and ensures that Gs aren’t idle when Ps are available.

Also, when a G has completed its moment on the P, then some LIFO behaviour takes place with that P’s runq, this ensures that CPU caches are kept warm, reducing how often the contents of caches being changed.

A P has an M (OS thread) that does all of the work. When a G notices that it is making a (potentially) blocking system call a new P is created, and the potentially blocking G is moved to the new P, makes its call which then blocks the M. The existing P works on other tasks in its runq, with a new M, or steals some work from the global runq or another Ps runq.

The Go runtime has five opportunites to call the scheduler.

1. When the Garbage Collector starts the world after a mark or sweep operation.
2. When the Go keyword is encountered.
3. When a synchronisation package library call is made (most often this is a channel being written or read to/from, but checking the state of locks, etc will also involve the scheduler).
4. When a (potentially) blocking runtime.Syscall is made.
5. As of Go 1.14.1 every 10 ms the scheduler is called, and checks for CPU bound Goroutines that might be hogging the CPU for a given thread.

Why is this important? A developer needs to be aware how their Goroutines are scheduled, and that the scheduler is not going to behave like a kernel scheduler (which is designed to accommodate multiple processes that do not know about one another).

There is no magic, if a developer writes a CPU bound Goroutine, then the developer is expected to know when to yield that Goroutine’s slot on an M with a call to runtime.Gosched in order to prevent the (possible) starvation of other Goroutines.

The developer is also responsible for prioritising their Goroutines, as the Go runtime does not prioritise the Goroutines at all, save for the LIFO behaviour to keep caches warm.

For reference, the Go scheduler is defined in runtime/proc.go, there is also a link in the code to a Design document. The Go garbage collector lives mostly in runtime/mgc.go. Further, it’s also helpful to have read Modern Operating Systems.

If you wish to contrast the Go scheduler against those of other languages, then I recommend this excellent blog post on Haskell’s thread blocking behaviour and the Loom Proposal, which is about bringing Green threads back to Java, via fibres.