'Handle goroutine termination and error handling via error group?

I am trying to read multiple files in parallel in such a way so that each go routine that is reading a file write its data to that channel, then have a single go-routine that listens to that channel and adds the data to the map. Here is my play.

Below is the example from the play:

package main

import (
    "fmt"
    "sync"
)

func main() {
    var myFiles = []string{"file1", "file2", "file3"}
    var myMap = make(map[string][]byte)
    dataChan := make(chan fileData, len(myFiles))
    wg := sync.WaitGroup{}
    defer close(dataChan)
    // we create a wait group of N
    wg.Add(len(myFiles))
    for _, file := range myFiles {
        // we create N go-routines, one per file, each one will return a struct containing their filename and bytes from
        // the file via the dataChan channel
        go getBytesFromFile(file, dataChan, &wg)
    }
    // we wait until the wait group is decremented to zero by each instance of getBytesFromFile() calling waitGroup.Done()
    wg.Wait()
    for i := 0; i < len(myFiles); i++ {
        // we can now read from the data channel N times.
        file := <-dataChan
        myMap[file.name] = file.bytes
    }
    fmt.Printf("%+v\n", myMap)
}

type fileData struct {
    name  string
    bytes []byte
}

// how to handle error from this method if reading file got messed up?
func getBytesFromFile(file string, dataChan chan fileData, waitGroup *sync.WaitGroup) {
    bytes := openFileAndGetBytes(file)
    dataChan <- fileData{name: file, bytes: bytes}
    waitGroup.Done()
}

func openFileAndGetBytes(file string) []byte {
    return []byte(fmt.Sprintf("these are some bytes for file %s", file))
}

Problem Statement

How can I use golang.org/x/sync/errgroup to wait on and handle errors from goroutines or if there is any better way like using semaphore? For example if any one of my go routine fails to read data from file then I want to cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller). And it should automatically waits for all the supplied go routines to complete successfully for success case.

I also don't want to spawn 100 go-routines if total number of files is 100. I want to control the parallelism if possible if there is any way.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source