Synchronization

This chapter is from the book

In Chapter 11, we explained how to use channels for passing data between goroutines. Then in Chapter 12, we discussed how to use thecontext¹package to manage the cancellation of goroutines. In this chapter, we cover the final part of concurrent programming: synchronization.

We show you how to wait for a number of goroutines to finish. We explainrace conditions,²how to find them using Go’s-race³flag, and how to fix them withsync.Mutex⁴andsync.RWMutex。⁵

Finally, we discuss how to usesync.Onceto ensure a function is only executed one time.

Waiting for Goroutines with a WaitGroup

Often, you might want to wait for a number of goroutines to finish before you continue your program. For example, you might want to spawn a number of goroutines to create a number of thumbnails of different sizes and wait for them all to complete before you continue.

The Problem

Consider Listing 13.1. We launch5new goroutines, each of which creates a thumbnail of a different size. We then wait for all of them to complete.

Listing 13.1 Launching Multiple Goroutines to Complete One Task

funcTest_ThumbnailGenerator(t*testing。T) {t。Parallel()// image that we need thumbnails forconstimage="foo.png"// start 5 goroutines to generate thumbnailsfori:=0;i<5;i++ {// start a new goroutine for each thumbnailgogenerateThumbnail(image,i+1)}fmt。Println("Waiting for thumbnails to be generated") }

ThegenerateThumbnailfunction, Listing 13.2, generates a thumbnail of the specified size. In this example, we sleep one millisecond per “size” of thumbnail to simulate the time it takes to generate the thumbnail. For example, if we callgenerateThumbnail("foo.png", 200), we sleep 200 milliseconds before returning.

Listing 13.2 A Test Exiting before All Goroutines Have Finished

funcgenerateThumbnail(imagestring,sizeint) {// thumbnail to be generatedthumb:=fmt。Sprintf("%s@%dx.png",image,size)fmt。Println("Generating thumbnail:",thumb)// wait for the thumbnail to be readytime。Sleep(time。Millisecond*time。Duration(size))fmt。Println("Finished generating thumbnail:",thumb) }

---

$ go test -v === RUN Test_ThumbnailGenerator === PAUSE Test_ThumbnailGenerator === CONT Test_ThumbnailGenerator Waiting for thumbnails to be generated --- PASS: Test_ThumbnailGenerator (0.00s) PASS ok demo 0.408s

---

Go Version: go1.19

As you can see from the test output in Listing 13.2, the test exits before the thumbnails are generated.

Our tests exit prematurely because we have not provided any mechanics to ensure that we wait for all of the thumbnail goroutines to finish before we continue.

Using a WaitGroup

To help us solve this problem, we can use async.WaitGroup,⁶Listing 13.3, to track how many goroutines are still running and notify us when they have all finished.

Listing 13.3 Thesync.WaitGroupType

$ go doc -short sync.WaitGroup type WaitGroup struct { // Has unexported fields. } A WaitGroup waits for a collection of goroutines to finish. The maingoroutine calls Add to set the number of goroutines to wait for. Theneach of the goroutines runs and calls Done when finished. At the same时间,等待可以用来阻止,直到所有goroutines have finished. A WaitGroup must not be copied after first use. func (wg *WaitGroup) Add(delta int) func (wg *WaitGroup) Done() func (wg *WaitGroup) Wait()

---

Go Version: go1.19

The principle is simple: We create async.WaitGroupand use thesync.WaitGroup.Add⁷method to add to thesync.WaitGroupfor each goroutine we want to wait for. When we want to wait for all of the goroutines to finish, we call thesync.WaitGroup.Wait⁸method. When a goroutine finishes, it calls thesync.WaitGroup.Done⁹method to indicate that the goroutine is finished.

TheWaitMethod

As the name suggests, async.WaitGroupis about waiting for a group of tasks, or goroutines, to finish. To do this, we need a way of blocking until all of the tasks have finished. Thesync.WaitGroup.Waitmethod in Listing 13.4 does exactly that.

Thesync.WaitGroup.Waitmethod blocks until its internal counter is zero. When the counter is zero, it means that all of the tasks have finished, and we can unblock and continue.

Listing 13.4 Thesync.WaitGroup.WaitMethod

$ go doc sync.WaitGroup.Wait package sync // import "sync" func (wg *WaitGroup) Wait() Wait blocks until the WaitGroup counter is zero.

---

Go Version: go1.19

TheAddMethod

For async.WaitGroupto know how many goroutines it needs to wait for, we need to add them to thesync.WaitGroupusing thesync.WaitGroup.Addmethod, Listing 13.5.

Listing 13.5 Thesync.WaitGroup.AddMethod

$ go doc sync.WaitGroup.Add package sync // import "sync" func (wg *WaitGroup) Add(delta int) Add adds delta, which may be negative, to the WaitGroup counter.If the counter becomes zero, all goroutines blocked on Wait are released.If the counter goes negative, Add panics. Note that calls with a positive delta that occur when the counter iszero must happen before a Wait. Calls with a negative delta, or callswith a positive delta that start when the counter is greater than zero,may happen at any time. Typically this means the calls to Add shouldexecute before the statement creating the goroutine or other event to bewaited for. If a WaitGroup is reused to wait for several independentsets of events, new Add calls must happen after all previous Wait callshave returned. See the WaitGroup example.

---

Go Version: go1.19

Thesync.WaitGroup.Addmethod takes a single integer argument, which is the number of goroutines to wait for. There are, however, some caveats to be aware of.

Adding a Positive Number

Thesync.WaitGroup.Addmethod accepts anintargument, which is the number of goroutines to wait for. If we pass a positive number, thesync.WaitGroup.Addmethod adds that number of goroutines to thesync.WaitGroup。

As you can see from the test output in Listing 13.6, thesync.WaitGroup.Waitmethod blocks until the internal counter of thesync.WaitGroupreaches zero.

Listing 13.6 Adding a Positive Number of Goroutines

funcTest_WaitGroup_Add_Positive(t*testing。T) {t。Parallel()varcompletedbool// create a new waitgroup (count: 0)varwg sync。WaitGroup// add one to the waitgroup (count: 1)wg。Add(1)// launch a goroutine to call the Done() methodgo func(wg*sync。WaitGroup) {// sleep for a bittime。Sleep(time。Millisecond*10)fmt。Println("done with waitgroup")completed=true// call the Done() method to decrement // the waitgroup counter (count: 0)wg。Done() }(&wg)fmt。Println("waiting for waitgroup to unblock")// wait for the waitgroup to unblock (count: 1)wg。Wait()// (count: 0)fmt。Println("waitgroup is unblocked")if!completed{t。Fatal("waitgroup is not completed") } }

---

$ go test -v -run Positive === RUN Test_WaitGroup_Add_Positive === PAUSE Test_WaitGroup_Add_Positive === CONT Test_WaitGroup_Add_Positive waiting for waitgroup to unblock done with waitgroup waitgroup is unblocked --- PASS: Test_WaitGroup_Add_Positive (0.01s) PASS ok demo 0.351s

---

Go Version: go1.19

Adding a Zero Number

It is legal to call thesync.WaitGroup.Addmethod with a zero number,0, Listing 13.7. In this case, thesync.WaitGroup.Addmethod does nothing. The call becomes a no-op.

Listing 13.7 Adding a Zero Number of Goroutines

funcTest_WaitGroup_Add_Zero(t*testing。T) {t。Parallel()// create a new waitgroup (count: 0)varwg sync。WaitGroup// add 0 to the waitgroup (count: 0)wg。Add(0)// (count: 0)fmt。Println("waiting for waitgroup to unblock")// wait for the waitgroup to unblock (count: 0) // will not block since the counter is already 0wg。Wait()// (count: 0)fmt。Println("waitgroup is unblocked") }

---

$ go test -v -run Zero === RUN Test_WaitGroup_Add_Zero === PAUSE Test_WaitGroup_Add_Zero === CONT Test_WaitGroup_Add_Zero waiting for waitgroup to unblock waitgroup is unblocked --- PASS: Test_WaitGroup_Add_Zero (0.00s) PASS ok demo 0.166s

---

Go Version: go1.19

As you can see from the test output in Listing 13.7, thesync.WaitGroup.Waitmethod unblocked immediately because its internal counter is already zero.

Adding a Negative Number

When calling thesync.WaitGroup.Addmethod with a negative number, thesync.WaitGroup.Addmethod panics.

As you can see from the test output in Listing 13.8, thesync.WaitGroup.Waitmethod was never reached because thesync.WaitGroup.Addmethod panicked when we tried to add a negative number of goroutines.

Listing 13.8 Adding a Negative Number of Goroutines

funcTest_WaitGroup_Add_Negative(t*testing。T) {t。Parallel()// create a new waitgroup (count: 0)varwg sync。WaitGroup// use an anonymous function to trap the panic // so we can properly mark the test as a failurefunc() {// defer a function to catch the panicdefer func() {// recover the panicifr:=recover();r!=nil{// mark the test as a failuret。Fatal(r) } }()// add a negative number to the waitgroup // this will panic since the counter cannot be negativewg。Add(-1)fmt。Println("waiting for waitgroup to unblock")// this will never be reachedwg。Wait()fmt。Println("waitgroup is unblocked") }()}

---

$ go test -v -run Negative === RUN Test_WaitGroup_Add_Negative === PAUSE Test_WaitGroup_Add_Negative === CONT Test_WaitGroup_Add_Negative add_test.go:92: sync: negative WaitGroup counter --- FAIL: Test_WaitGroup_Add_Negative (0.00s) FAIL exit status 1 FAIL demo 0.753s

---

Go Version: go1.19

The Done Method

Once we increase that counter by calling thesync.WaitGroup.Addmethod, thesync.WaitGroup.Waitmethod blocks until we decrement the counter as we finish with each goroutine.

For each item we add to thesync.WaitGroupwith thesync.WaitGroup.Addmethod, we need to call thesync.WaitGroup.Donemethod, Listing 13.9, to indicate that the goroutine is finished.

Listing 13.9 Thesync.WaitGroup.Donemethod

$ go doc sync.WaitGroup.Done package sync // import "sync" func (wg *WaitGroup) Done() Done decrements the WaitGroup counter by one.

---

Go Version: go1.19

Consider Listing 13.10, which createsNgoroutines and addsNto thesync.WaitGroupusing thesync.WaitGroup.Addmethod. Each goroutine calls thesync.WaitGroup.Donemethod after it finishes. We then use thesync.WaitGroup.Waitmethod to wait for all of the goroutines to finish.

Listing 13.10 Testing thesync.WaitGroup.DoneMethod

funcTest_WaitGroup_Done(t*testing。T) {t。Parallel()constN=5// create a new waitgroup (count: 0)varwg sync。WaitGroup// add 5 to the waitgroup (count: 5)wg。Add(N)fori:=0;i<N;i++ {// launch a goroutine that will call the // waitgroup's Done method when it finishesgo func(iint) {// sleep brieflytime。Sleep(time。Millisecond*time。Duration(i))fmt。Println("decrementing waiting by 1")// call the waitgroup's Done method // (count: count - 1)wg。Done()}(i+1) }fmt。Println("waiting for waitgroup to unblock")wg。Wait()fmt。Println("waitgroup is unblocked") }

---

美元去测试- v超时1 s = = = Test_WaitGroup_Do运行ne === PAUSE Test_WaitGroup_Done === CONT Test_WaitGroup_Done waiting for waitgroup to unblock decremeting waiting by 1 decremeting waiting by 1 decremeting waiting by 1 decremeting waiting by 1 decremeting waiting by 1 waitgroup is unblocked --- PASS: Test_WaitGroup_Done (0.01s) PASS ok demo 0.384s

---

Go Version: go1.19

As we can see from the test output, Listing 13.10, thesync.WaitGroup.Waitmethod unblocked after all of the goroutines finished.

Improper Usage

If you don’t callsync.WaitGroup.Doneexactly once for each item you add withsync.WaitGroup.Add,sync.WaitGroup.Waitmethod will block forever, which causes a deadlock and crashes your program, as shown in Listing 13.11.

Listing 13.11 Decrementing async.WaitGroupwith thesync.WaitGroup.DoneMethod

funcTest_WaitGroup_Done(t*testing。T) {t。Parallel()constN=5// create a new waitgroup (count: 0)varwg sync。WaitGroup// add 5 to the waitgroup (count: 5)wg。Add(N)fori:=0;i<N;i++ {// launch a goroutine that will call the // waitgroup's Done method when it finishesgo func(iint) {// sleep brieflytime。Sleep(time。Millisecond*time。Duration(i))fmt。Println("finished")// exiting with calling the Done method // (count: count)}(i+1) }fmt。Println("waiting for waitgroup to unblock")// this will never unblock // because the goroutines never call Done // and the application will deadlock and panicwg。Wait()fmt。Println("waitgroup is unblocked")}

---

美元去测试- v超时1 s = = = Test_WaitGroup_Do运行ne === PAUSE Test_WaitGroup_Done === CONT Test_WaitGroup_Done waiting for waitgroup to unblock finished finished finished finished finished panic: test timed out after 1s goroutine 19 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2029 +0x8c created by time.goFunc /usr/local/go/src/time/sleep.go:176 +0x3c goroutine 1 [chan receive]: testing.tRunner.func1() /usr/local/go/src/testing/testing.go:1405 +0x45c testing.tRunner(0x140001361a0, 0x1400010fcb8) /usr/local/go/src/testing/testing.go:1445 +0x14c testing.runTests(0x1400001e280?, {0x101045ea0, 0x1, 0x1},{0x6e00000000000000?, 0x100e71218?, 0x10104e640?}) /usr/local/go/src/testing/testing.go:1837 +0x3f0 testing.(*M).Run(0x1400001e280) /usr/local/go/src/testing/testing.go:1719 +0x500 main.main() _testmain.go:47 +0x1d0 goroutine 4 [semacquire]: sync.runtime_Semacquire(0x0?) /usr/local/go/src/runtime/sema.go:56 +0x2c sync.(*WaitGroup).Wait(0x14000012140) /usr/local/go/src/sync/waitgroup.go:136 +0x88 demo.Test_WaitGroup_Done(0x0?) ./done_test.go:43 0xd0 testing.tRunner(0x14000136340, 0x100fa1580) /usr/local/go/src/testing/testing.go:1439 +0x110 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1486 +0x300 exit status 2 FAIL demo 1.225s

---

Go Version: go1.19

If you callsync.WaitGroup.Donemore than the number of items you added withsync.WaitGroup.Add,sync.WaitGroup.Donemethod panics, Listing 13.12. The result is the same as if you calledsync.WaitGroup.Addwith a negative number.

Listing 13.12 Panicking from Decrementingsync.WaitGroupToo Many Times

funcTest_WaitGroup_Done(t*testing。T) {t。Parallel()func() {// defer a function to catch the panicdefer func() {// recover the panicifr:=recover();r!=nil{// mark the test as a failuret。Fatal(r) } }()// create a new waitgroup (count: 0)varwg sync。WaitGroup// call done creating a negative // waitgroup counterwg。Done()// this line is never reachedfmt。Println("waitgroup is unblocked") }()}

---

美元去测试- v超时1 s = = = Test_WaitGroup_Do运行ne === PAUSE Test_WaitGroup_Done === CONT Test_WaitGroup_Done done_test.go:20: sync: negative WaitGroup counter --- FAIL: Test_WaitGroup_Done (0.00s) FAIL exit status 1 FAIL demo 0.416s

---

Go Version: go1.19

Wrapping Up Wait Groups

Using async.WaitGroupis a great way to manage the number of goroutines or any other number of tests that need to finish before your program can continue.

As you can see, we can effectively use async.WaitGroupto manage the thumbnail generator goroutines from our initial example.

In Listing 13.13, we create a newsync.WaitGroup。Then, in theforloop, we use thesync.WaitGroup.Addmethod to add1to thesync.WaitGroup。We then pass a pointer to thegenerateThumbnailfunction tosync.WaitGroup。A pointer is needed because thegenerateThumbnailfunction needs to be able to modify thesync.WaitGroupby calling thesync.WaitGroup.Donemethod.

Finally, we call thesync.WaitGroup.Waitmethod to wait for all of the goroutines to finish.

Listing 13.13 Using async.WaitGroupto Manage the Thumbnail Generator Goroutines

funcTest_ThumbnailGenerator(t*testing。T) {t。Parallel()// image that we need thumbnails forconstimage="foo.png"varwg sync。WaitGroup// start 5 goroutines to generate thumbnailsfori:=0;i<5;i++ {wg。Add(1)// start a new goroutine for each thumbnailgogenerateThumbnail(&wg,image,i+1)}fmt。Println("Waiting for thumbnails to be generated")// wait for all goroutines to finishwg。Wait()fmt。Println("Finished generate all thumbnails") }

ThegenerateThumbnailfunction now receives a pointer to thesync.WaitGroupand defers a call to thesync.WaitGroup.Donemethod to indicate that the goroutine is finished when the function exits.

Finally, as you can see from our test output in Listing 13.14, the application now finishes successfully.

Listing 13.14 Generating Thumbnails Using async.WaitGroup

funcgenerateThumbnail(wg*sync。WaitGroup,imagestring,sizeint) {deferwg。Done()// thumbnail to be generatedthumb:=fmt。Sprintf("%s@%dx.png",image,size)fmt。Println("Generating thumbnail:",thumb)// wait for the thumbnail to be readytime。Sleep(time。Millisecond*time。Duration(size))fmt。Println("Finished generating thumbnail:",thumb) }

---

$ go test -v === RUN Test_ThumbnailGenerator === PAUSE Test_ThumbnailGenerator === CONT Test_ThumbnailGenerator Waiting for thumbnails to be generated Generating thumbnail: foo.png@5x.png Generating thumbnail: foo.png@3x.png Generating thumbnail: foo.png@4x.png Generating thumbnail: foo.png@2x.png Generating thumbnail: foo.png@1x.png Finished generating thumbnail: foo.png@1x.png Finished generating thumbnail: foo.png@2x.png Finished generating thumbnail: foo.png@3x.png Finished generating thumbnail: foo.png@4x.png Finished generating thumbnail: foo.png@5x.png Finished generate all thumbnails --- PASS: Test_ThumbnailGenerator (0.01s) PASS ok demo 0.310s

---

Go Version: go1.19