By user1243746


2012-05-28 08:26:00 8 Comments

How can I pipe several external commands together in Go? I've tried this code but I get an error that says exit status 1.

package main

import (
    "io"
    "log"
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    stdout1, err := c1.StdoutPipe()
    if err != nil {
        log.Fatal(err)
    }

    if err = c1.Start(); err != nil {
        log.Fatal(err)
    }
    if err = c1.Wait(); err != nil {
        log.Fatal(err)
    }

    c2 := exec.Command("wc", "-l")
    c2.Stdin = stdout1

    stdout2, err := c2.StdoutPipe()
    if err != nil {
        log.Fatal(err)
    }

    if err = c2.Start(); err != nil {
        log.Fatal(err)
    }
    if err = c2.Wait(); err != nil {
        log.Fatal(err)
    }

    io.Copy(os.Stdout, stdout2)
}

6 comments

@eriel marimon 2018-09-30 17:38:43

package main

import (
    ...
    pipe "github.com/b4b4r07/go-pipe"
)

func main() {
    var b bytes.Buffer
    pipe.Command(&b,
        exec.Command("ls", "/Users/b4b4r07/Downloads"),
        exec.Command("grep", "Vim"),
    )

    io.Copy(os.Stdout, &b)
}

I spent a good day trying to use Denys S├ęguret answer to come up with a wrapper for multiple exec.Command before I came across this neat package by b4b4r07.

@eriel marimon 2018-09-30 17:44:28

I just realized the implementation of this package is the same as the answer by @Tyndyll up above. Just noting...

@WeakPointer 2016-03-12 15:38:42

Like the first answer but with the first command started and waited for in a goroutine. This keeps the pipe happy.

package main

import (
    "io"
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    c2 := exec.Command("wc", "-l")

    pr, pw := io.Pipe()
    c1.Stdout = pw
    c2.Stdin = pr
    c2.Stdout = os.Stdout

    c1.Start()
    c2.Start()

    go func() {
        defer pw.Close()

        c1.Wait()
    }()
    c2.Wait()
}

@Jason Stewart 2018-07-26 18:29:54

It would probably work just fine without the goroutine if it used os.Pipe() instead of io.Pipe(). Let the OS do the byte-shuffling by itself.

@WeakPointer 2019-02-10 15:22:11

@JasonStewart This advice seems to be correct, thx. I've begun using it with no ill effects so far.

@Tyndyll 2014-10-24 05:08:43

This is a fully working example. The Execute function takes any number of exec.Cmd instances (using a variadic function) and then loops over them correctly attaching the output of stdout to the stdin of the next command. This must be done before any function is called.

The call function then goes about calling the commands in a loop, using defers to call recursively and ensuring proper closure of pipes

package main

import (
    "bytes"
    "io"
    "log"
    "os"
    "os/exec"
)

func Execute(output_buffer *bytes.Buffer, stack ...*exec.Cmd) (err error) {
    var error_buffer bytes.Buffer
    pipe_stack := make([]*io.PipeWriter, len(stack)-1)
    i := 0
    for ; i < len(stack)-1; i++ {
        stdin_pipe, stdout_pipe := io.Pipe()
        stack[i].Stdout = stdout_pipe
        stack[i].Stderr = &error_buffer
        stack[i+1].Stdin = stdin_pipe
        pipe_stack[i] = stdout_pipe
    }
    stack[i].Stdout = output_buffer
    stack[i].Stderr = &error_buffer

    if err := call(stack, pipe_stack); err != nil {
        log.Fatalln(string(error_buffer.Bytes()), err)
    }
    return err
}

func call(stack []*exec.Cmd, pipes []*io.PipeWriter) (err error) {
    if stack[0].Process == nil {
        if err = stack[0].Start(); err != nil {
            return err
        }
    }
    if len(stack) > 1 {
        if err = stack[1].Start(); err != nil {
             return err
        }
        defer func() {
            if err == nil {
                pipes[0].Close()
                err = call(stack[1:], pipes[1:])
            }
        }()
    }
    return stack[0].Wait()
}

func main() {
    var b bytes.Buffer
    if err := Execute(&b,
        exec.Command("ls", "/Users/tyndyll/Downloads"),
        exec.Command("grep", "as"),
        exec.Command("sort", "-r"),
    ); err != nil {
        log.Fatalln(err)
    }
    io.Copy(os.Stdout, &b)
}

Available in this gist

https://gist.github.com/tyndyll/89fbb2c2273f83a074dc

A good point to know is that shell variables like ~ are not interpolated

@Tyndyll 2015-05-27 07:45:16

Updated - in my defense I'd answered it at 5am after a few hours working on it :)

@eriel marimon 2018-09-30 17:46:20

@Denys Séguret 2012-05-28 08:31:54

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts. The pipe will be closed automatically after Wait sees the command exit.

(from http://golang.org/pkg/os/exec/#Cmd.StdinPipe )

The fact you do c1.Wait closes the stdoutPipe.

I made a working example (just a demo, add error catching!) :

package main

import (
    "bytes"
    "io"
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    c2 := exec.Command("wc", "-l")

    r, w := io.Pipe() 
    c1.Stdout = w
    c2.Stdin = r

    var b2 bytes.Buffer
    c2.Stdout = &b2

    c1.Start()
    c2.Start()
    c1.Wait()
    w.Close()
    c2.Wait()
    io.Copy(os.Stdout, &b2)
}

@user1243746 2012-05-28 10:02:34

Why do use io.Pipe rather than exec.Cmd.StdoutPipe?

@WeakPointer 2016-03-12 15:25:00

I like io.Pipe too, but putting the c1 start into a separate goroutine works better for me. See my modified version below.

@Stefan Saru 2015-05-19 15:04:02

For simple scenarios, you could use this approach:

bash -c "echo 'your command goes here'"

For example, this function retrieves the CPU model name using piped commands:

func getCPUmodel() string {
        cmd := "cat /proc/cpuinfo | egrep '^model name' | uniq | awk '{print substr($0, index($0,$4))}'"
        out, err := exec.Command("bash","-c",cmd).Output()
        if err != nil {
                return fmt.Sprintf("Failed to execute command: %s", cmd)
        }
        return string(out)
}

@henrywallace 2019-01-31 19:45:26

It should be noted, however, that this will fail if the cmd is too long. In particular if it's over 131072 bytes long, then you'll probably get something like fork/exec /bin/bash: argument list too long, see here. In this case you might have change or divide up your command, or resort to the more hefty io.Pipe methods listed elsewhere in this question's answers.

@Matt 2012-06-08 17:08:10

package main

import (
    "os"
    "os/exec"
)

func main() {
    c1 := exec.Command("ls")
    c2 := exec.Command("wc", "-l")
    c2.Stdin, _ = c1.StdoutPipe()
    c2.Stdout = os.Stdout
    _ = c2.Start()
    _ = c1.Run()
    _ = c2.Wait()
}

@Anthony Hunt 2014-10-05 18:06:10

I'm basically using the same code but often I get a "broken pipe" error. Any idea what could cause that? stackoverflow.com/q/26122072/4063955

@RickyA 2014-10-06 10:23:01

@AnthonyHat: please put this comment on your new question, so we can see you saw this one and it didn't work for you.

@Matt 2014-10-06 22:13:20

Broken pipe occurs when one process tries to write into a pipe but the other side of the pipe has already been closed. For example, if the "wc -l" exited before the "ls" finished in the above example, the "ls" would get a Broken Pipe error/signal.

@user7044 2015-02-22 19:41:41

How can i make this program concurrent, since there is io happening here(stdin/stdout)!!

@Matt 2015-05-19 14:50:02

@user7044, I'm not sure what you mean. The two commands "ls" and "wc -l" are running concurrently in this example, with the output from ls being piped to wc, which can start reading the output from ls before ls finishes writing all of it.

Related Questions

Sponsored Content

3 Answered Questions

[SOLVED] Does it make sense to have two packages in the same directory?

  • 2013-12-06 15:40:17
  • xrash
  • 26717 View
  • 26 Score
  • 3 Answer
  • Tags:   go package

12 Answered Questions

[SOLVED] How to pipe stderr, and not stdout?

7 Answered Questions

[SOLVED] How can I redirect and append both stdout and stderr to a file with Bash?

1 Answered Questions

[SOLVED] using ioctl to fill a winsize struct when piping to stdin

  • 2018-09-13 16:06:11
  • Chris
  • 133 View
  • -1 Score
  • 1 Answer
  • Tags:   c pipe tty ioctl

1 Answered Questions

[SOLVED] Implementing shell in C - pipelined input has correct output but exits loop

  • 2018-06-09 12:04:26
  • user9917989
  • 51 View
  • 0 Score
  • 1 Answer
  • Tags:   c shell pipe fork dup2

1 Answered Questions

[SOLVED] How to copy a file from one directory to another using "os/exec" package in GO

  • 2017-08-31 19:42:19
  • nchauhan503
  • 180 View
  • -9 Score
  • 1 Answer
  • Tags:   linux go

2 Answered Questions

[SOLVED] access golang websocket server with nodejs client

1 Answered Questions

[SOLVED] Specific pipe command in Ubuntu's shell handling in C

1 Answered Questions

[SOLVED] golang tcp socket can't close after get File()

  • 2015-03-10 15:29:25
  • vzex
  • 964 View
  • 0 Score
  • 1 Answer
  • Tags:   sockets tcp go

2 Answered Questions

[SOLVED] Read / Write through a pipe in C

  • 2013-02-13 21:17:23
  • markmb
  • 14075 View
  • 3 Score
  • 2 Answer
  • Tags:   c fork pipe

Sponsored Content