By user2668128


2014-02-26 17:46:52 8 Comments

I'm thinking of using Docker to build my dependencies on a Continuous Integration (CI) server, so that I don't have to install all the runtimes and libraries on the agents themselves.

To achieve this I would need to copy the build artifacts that are built inside the container back into the host. Is that possible?

17 comments

@John Drinane 2020-01-14 23:43:51

This can also be done in the SDK for example python. If you already have a container built you can lookup the name via console ( docker ps -a ) name seems to be some concatenation of a scientist and an adjective (i.e. "relaxed_pasteur").

Check out help(container.get_archive) :

Help on method get_archive in module docker.models.containers:

get_archive(path, chunk_size=2097152) method of docker.models.containers.Container instance
    Retrieve a file or folder from the container in the form of a tar
    archive.

    Args:
        path (str): Path to the file or folder to retrieve
        chunk_size (int): The number of bytes returned by each iteration
            of the generator. If ``None``, data will be streamed as it is
            received. Default: 2 MB

    Returns:
        (tuple): First element is a raw tar data stream. Second element is
        a dict containing ``stat`` information on the specified ``path``.

    Raises:
        :py:class:`docker.errors.APIError`
            If the server returns an error.

    Example:

        >>> f = open('./sh_bin.tar', 'wb')
        >>> bits, stat = container.get_archive('/bin/sh')
        >>> print(stat)
        {'name': 'sh', 'size': 1075464, 'mode': 493,
         'mtime': '2018-10-01T15:37:48-07:00', 'linkTarget': ''}
        >>> for chunk in bits:
        ...    f.write(chunk)
        >>> f.close()

So then something like this will pull out from the specified path ( /output) in the container to your host machine and unpack the tar.

import docker
import os
import tarfile

# Docker client
client = docker.from_env()
#container object
container = client.containers.get("relaxed_pasteur")
#setup tar to write bits to
f = open(os.path.join(os.getcwd(),"output.tar"),"wb")
#get the bits
bits, stat = container.get_archive('/output')
#write the bits
for chunk in bits:
    f.write(chunk)
f.close()
#unpack
tar = tarfile.open("output.tar")
tar.extractall()
tar.close()

@BMitch 2019-11-07 15:48:54

With the release of Docker 19.03, you can skip creating the container and even building an image. There's an option with BuildKit based builds to change the output destination. You can use this to write the results of the build to your local directory rather than into an image. E.g. here's a build of a go binary:

$ ls
Dockerfile  go.mod  main.go

$ cat Dockerfile
FROM golang:1.12-alpine as dev
RUN apk add --no-cache git ca-certificates
RUN adduser -D appuser
WORKDIR /src
COPY . /src/
CMD CGO_ENABLED=0 go build -o app . && ./app

FROM dev as build
RUN CGO_ENABLED=0 go build -o app .
USER appuser
CMD [ "./app" ]

FROM scratch as release
COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /src/app /app
USER appuser
CMD [ "/app" ]

FROM scratch as artifact
COPY --from=build /src/app /app

FROM release

From the above Dockerfile, I'm building the artifact stage that only includes the files I want to export. And the newly introduced --output flag lets me write those to a local directory instead of an image. This needs to be performed with the BuildKit engine that ships with 19.03:

$ DOCKER_BUILDKIT=1 docker build --target artifact --output type=local,dest=. .
[+] Building 43.5s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                              0.7s
 => => transferring dockerfile: 572B                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                 0.5s
 => => transferring context: 2B                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/golang:1.12-alpine                                                             0.9s
 => [dev 1/5] FROM docker.io/library/golang:[email protected]:50deab916cce57a792cd88af3479d127a9ec571692a1a9c22109532c0d0499a0  22.5s
 => => resolve docker.io/library/golang:[email protected]:50deab916cce57a792cd88af3479d127a9ec571692a1a9c22109532c0d0499a0       0.0s
 => => sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd 155B / 155B                                        0.3s
 => => sha256:50deab916cce57a792cd88af3479d127a9ec571692a1a9c22109532c0d0499a0 1.65kB / 1.65kB                                    0.0s
 => => sha256:2ecd820bec717ec5a8cdc2a1ae04887ed9b46c996f515abc481cac43a12628da 1.36kB / 1.36kB                                    0.0s
 => => sha256:6a17089e5a3afc489e5b6c118cd46eda66b2d5361f309d8d4b0dcac268a47b13 3.81kB / 3.81kB                                    0.0s
 => => sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17 2.79MB / 2.79MB                                    0.6s
 => => sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5 301.72kB / 301.72kB                                0.4s
 => => sha256:025f14a3d97f92c07a07446e7ea8933b86068d00da9e252cf3277e9347b6fe69 125.33MB / 125.33MB                               13.7s
 => => sha256:7047deb9704134ff71c99791be3f6474bb45bc3971dde9257ef9186d7cb156db 125B / 125B                                        0.8s
 => => extracting sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17                                         0.2s
 => => extracting sha256:8ef94372a977c02d425f12c8cbda5416e372b7a869a6c2b20342c589dba3eae5                                         0.1s
 => => extracting sha256:1ec62c064901392a6722bb47a377c01a381f4482b1ce094b6d28682b6b6279fd                                         0.0s
 => => extracting sha256:025f14a3d97f92c07a07446e7ea8933b86068d00da9e252cf3277e9347b6fe69                                         5.2s
 => => extracting sha256:7047deb9704134ff71c99791be3f6474bb45bc3971dde9257ef9186d7cb156db                                         0.0s
 => [internal] load build context                                                                                                 0.3s
 => => transferring context: 2.11kB                                                                                               0.0s
 => [dev 2/5] RUN apk add --no-cache git ca-certificates                                                                          3.8s
 => [dev 3/5] RUN adduser -D appuser                                                                                              1.7s
 => [dev 4/5] WORKDIR /src                                                                                                        0.5s
 => [dev 5/5] COPY . /src/                                                                                                        0.4s
 => [build 1/1] RUN CGO_ENABLED=0 go build -o app .                                                                              11.6s
 => [artifact 1/1] COPY --from=build /src/app /app                                                                                0.5s
 => exporting to client                                                                                                           0.1s
 => => copying files 10.00MB                                                                                                      0.1s

After the build was complete the app binary was exported:

$ ls
Dockerfile  app  go.mod  main.go

$ ./app
Ready to receive requests on port 8080

Docker has other options to the --output flag documented in their upstream BuildKit repo: https://github.com/moby/buildkit#output

@burtsevyg 2019-12-10 17:15:34

standard build cache not used for build with output, it's bad

@BMitch 2019-12-10 18:34:48

@burtsevyg Buildkit is a different builder, using a different cache environment. It's much more cache efficient.

@burtsevyg 2019-12-10 21:10:09

thank you I will improve my build nodes.

@creack 2014-02-26 18:31:40

In order to copy a file from a container to the host, you can use the command

docker cp <containerId>:/file/path/within/container /host/path/target

Here's an example:

$ sudo docker cp goofy_roentgen:/out_read.jpg .

Here goofy_roentgen is the container name I got from the following command:

$ sudo docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                            NAMES
1b4ad9311e93        bamos/openface      "/bin/bash"         33 minutes ago      Up 33 minutes       0.0.0.0:8000->8000/tcp, 0.0.0.0:9000->9000/tcp   goofy_roentgen

You can also use (part of) the Container ID. The following command is equivalent to the first

$ sudo docker cp 1b4a:/out_read.jpg .

@Josh Habdas 2015-06-03 15:29:46

Here's a handy way to get at your latest container if you're simply using docker for a temp Linux environment: docker ps -alq.

@ecoe 2015-12-30 18:45:39

this cp command works as-is for copying directory trees as well (not just a single file).

@Robin Loxley 2016-02-19 08:58:00

This solution works in case for non-root docker user, who can't sudo in docker.

@Freedom_Ben 2016-06-18 21:01:51

In newer versions of docker you can copy bidirectionally (host to container or container to host) with docker cp ...

@Dipak 2016-06-21 06:44:31

I am getting following error : Cannot connect to the Docker daemon. Is the docker daemon running on this host? !!!

@Yonatan 2016-06-22 14:41:01

@DipakChandranP try adding sudo (sudo docker cp ...) if your user is not permitted to do docker stuff. Happens to me all the time.

@Dipak 2016-06-23 06:19:21

@Yonatan Got it resolved, I wasn't in docker terminal(using mac) that was the reason. Thanks man.

@Harrison Powers 2016-07-26 19:07:02

I needed docker cp -L to copy symlinks

@haridsv 2016-07-29 12:48:01

As of docker 1.11.2, if you copy from one container to another directly, you get the error message copying between containers is not supported.

@Gnana 2016-12-01 21:56:40

My Jenkins running as container. it needs to copy WAR to host folder. how do to this one ? any help

@Martlark 2017-05-24 02:08:25

NOTE: the container does not have to be running to use the cp command. Handy if your container constantly crashes.

@GiM 2017-07-26 11:28:55

apparently this does not work if I specify remote container path like: foo:port/container:tag

@creack 2017-07-29 03:55:28

AFAIK there is no such thing as remote container. I think you are talking about remote image (i.e. we start a container runtime from a static image), in which case indeed, it wont work. You need to use the container (runtime, local) name or ID.

@ColinWa 2017-10-25 12:27:14

when i include a command like the one in this answer, into a .sh file alongside other linux commands, it fails when executing the docker command. Any ideas

@Bej 2017-10-26 05:46:26

What if I want to copy the file from inside the container environment?

@Benyamin Jafari 2019-03-12 12:35:13

@creack Is there a manner which copies a file from the container to the host, into the container ('docker exec -it <container-name> bash' then send a file to the host)?

@Mohammad Masoumi 2019-10-14 07:43:49

It's better to use the last version of docker commands. docker container ...

@Astariul 2019-10-15 00:23:26

I couldn't copy a file using ~/path, I had to use the full path (/root/path) (you can get it with the pwd command)

@keshav 2019-11-20 07:13:55

getting this error Error response from daemon: error processing tar file: docker-tar: relocation error: /lib/x86_64-linux-gnu/libnss_files.so.2: symbol __libc_readline_unlocked, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

@Ishan Bhatt 2018-07-05 08:25:27

You do not need to use docker run

You can do it with docker create

From the docs The docker create command creates a writeable container layer over the specified image and prepares it for running the specified command. The container ID is then printed to STDOUT. This is similar to docker run -d except the container is never started.

So, you can do

docker create -ti --name dummy IMAGE_NAME bash
docker cp dummy:/path/to/file /dest/to/file
docker rm -f dummy

Here, you never start the container. That looked beneficial to me.

@Honza Kalfus 2019-01-25 12:06:14

This needs more upvotes. Great for when you just need to build something in a container and then copy the outputs.

@Mark 2019-07-03 07:52:53

@HonzaKalfus I agree this needs to be higher. This is exactly what I was after. I used this so that I could build some binary files using a known environment (amazon linux at a specific version). was able to make a shell script that fully built the docker and extracted the resultant binary from it! Perfect.

@matiasg 2019-08-24 11:41:34

I'm confused by the -fv options in the rm. You did not mount any volumes in the create part. Is the v really needed?

@Ishan Bhatt 2019-09-11 11:31:14

@matiasg Good catch, let me remove it, It's just the stuff I have been using.

@jII 2019-10-18 16:13:50

Is -ti required and bash required?

@Ishan Bhatt 2019-10-19 01:56:17

@jII, I had done it because later on, I do docker run on it. In simple cases, it is not needed but it doesn't harm here too.

@juzzlin 2019-12-12 13:53:30

Is it somehow possible to use wildcards? I mean...I don't know the exact name of the file I need to copy because it has a version number on it.

@rubicks 2017-05-07 15:54:32

TLDR;

$ docker run --rm -iv${PWD}:/host-volume my-image sh -s <<EOF
chown $(id -u):$(id -g) my-artifact.tar.xz
cp -a my-artifact.tar.xz /host-volume
EOF

Description

docker run with a host volume, chown the artifact, cp the artifact to the host volume:

$ docker build -t my-image - <<EOF
> FROM busybox
> WORKDIR /workdir
> RUN touch foo.txt bar.txt qux.txt
> EOF
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM busybox
 ---> 00f017a8c2a6
Step 2/3 : WORKDIR /workdir
 ---> Using cache
 ---> 36151d97f2c9
Step 3/3 : RUN touch foo.txt bar.txt qux.txt
 ---> Running in a657ed4f5cab
 ---> 4dd197569e44
Removing intermediate container a657ed4f5cab
Successfully built 4dd197569e44

$ docker run --rm -iv${PWD}:/host-volume my-image sh -s <<EOF
chown -v $(id -u):$(id -g) *.txt
cp -va *.txt /host-volume
EOF
changed ownership of '/host-volume/bar.txt' to 10335:11111
changed ownership of '/host-volume/qux.txt' to 10335:11111
changed ownership of '/host-volume/foo.txt' to 10335:11111
'bar.txt' -> '/host-volume/bar.txt'
'foo.txt' -> '/host-volume/foo.txt'
'qux.txt' -> '/host-volume/qux.txt'

$ ls -n
total 0
-rw-r--r-- 1 10335 11111 0 May  7 18:22 bar.txt
-rw-r--r-- 1 10335 11111 0 May  7 18:22 foo.txt
-rw-r--r-- 1 10335 11111 0 May  7 18:22 qux.txt

This trick works because the chown invocation within the heredoc the takes $(id -u):$(id -g) values from outside the running container; i.e., the docker host.

The benefits are:

  • you don't have to docker container run --name or docker container create --name before
  • you don't have to docker container rm after

@Marc Ghorayeb 2019-01-23 18:47:06

Upvoted for the comparison between cp and volume-based answers. Also, for the id trick for ownership, that is a real headache sometimes

@djhaskin987 2014-10-01 20:58:25

Mount a "volume" and copy the artifacts into there:

mkdir artifacts
docker run -i -v ${PWD}/artifacts:/artifacts ubuntu:14.04 sh << COMMANDS
# ... build software here ...
cp <artifact> /artifacts
# ... copy more artifacts into `/artifacts` ...
COMMANDS

Then when the build finishes and the container is no longer running, it has already copied the artifacts from the build into the artifacts directory on the host.

Edit

Caveat: When you do this, you may run into problems with the user id of the docker user matching the user id of the current running user. That is, the files in /artifacts will be shown as owned by the user with the UID of the user used inside the docker container. A way around this may be to use the calling user's UID:

docker run -i -v ${PWD}:/working_dir -w /working_dir -u $(id -u) \
    ubuntu:14.04 sh << COMMANDS
# Since $(id -u) owns /working_dir, you should be okay running commands here
# and having them work. Then copy stuff into /working_dir/artifacts .
COMMANDS

@Dimchansky 2015-03-30 15:21:42

Actually you can use chown command to match user id and group id on the host machine.

@djhaskin987 2018-06-19 16:08:23

@Frondor See volume config reference docs.docker.com/compose/compose-file/…

@Frondor 2018-06-19 23:46:23

Already did, and that won't work. Once the container copied files to the volume for first time, the next time, the volume is not empty anymore and the files are not being overridden by the newer ones. The container is giving priority to the host files (the ones copied the first time you mounted the container image).

@djhaskin987 2018-06-20 20:10:36

sounds like something that could be its own SO question @Frondor

@ToolmakerSteve 2019-04-03 10:52:21

@Frondor - are you able to rm the existing files first?

@zytfo 2019-01-23 16:13:08

You can use bind instead of volume if you want to mount only one folder, not create special storage for a container:

  1. Build your image with tag :

    docker build . -t <image>

  2. Run your image and bind current $(pwd) directory where app.py stores and map it to /root/example/ inside your container.

    docker run --mount type=bind,source="$(pwd)",target=/root/example/ <image> python app.py

@Khachornchit Songsaen 2018-11-06 09:38:35

I used PowerShell (Admin) with this command.

docker cp {container id}:{container path}/error.html  C:\\error.html

Example

docker cp ff3a6608467d:/var/www/app/error.html  C:\\error.html

@Chandra Pal 2018-06-29 10:45:21

Create a path where you want to copy the file and then use:

docker run -d -v hostpath:dockerimag

@cmcginty 2018-04-05 05:40:14

Most of the answers do not indicate that the container must run before docker cp will work:

docker build -t IMAGE_TAG .
docker run -d IMAGE_TAG
CONTAINER_ID=$(docker ps -alq)
# If you do not know the exact file name, you'll need to run "ls"
# FILE=$(docker exec CONTAINER_ID sh -c "ls /path/*.zip")
docker cp $CONTAINER_ID:/path/to/file .
docker stop $CONTAINER_ID

@ToolmakerSteve 2019-04-03 11:11:09

BTW, Whether the container must/may be running/stopped/either seems to depend on type of host/virtualization-technique. Current docker doc says "The CONTAINER can be a running or stopped container.". Multiple places on SO, including a comment on the accepted answer, say "this also works on a stopped container". Under Windows Hyper-V, it is apparently necessary to stop container before copying a file.

@cancerbero 2017-11-09 23:06:01

If you don't have a running container, just an image, and assuming you want to copy just a text file, you could do something like this:

docker run the-image cat path/to/container/file.txt > path/to/host/file.txt

@shuaihanhungry 2018-05-16 02:55:20

@s g 2017-12-30 04:06:37

If you just want to pull a file from an image (instead of a running container) you can do this:

docker run --rm <image> cat <source> > <local_dest>

This will bring up the container, write the new file, then remove the container. One drawback, however, is that the file permissions and modified date will not be preserved.

@Innocent Anigbo 2017-05-16 16:29:31

Create a data directory on the host system (outside the container) and mount this to a directory visible from inside the container. This places the files in a known location on the host system, and makes it easy for tools and applications on the host system to access the files

docker run -d -v /path/to/Local_host_dir:/path/to/docker_dir docker_image:tag

@BMitch 2017-05-16 16:31:47

That lets you inject a directory and it's contents from the host into the container. It doesn't let you copy files from the container back out to the host.

@giorgiosironi 2017-12-19 14:42:50

It does if the host folder has very wide permissions?

@Paul 2017-01-17 08:10:44

I am posting this for anyone that is using Docker for Mac. This is what worked for me:

 $ mkdir mybackup # local directory on Mac

 $ docker run --rm --volumes-from <containerid> \
    -v `pwd`/mybackup:/backup \  
    busybox \                   
    cp /data/mydata.txt /backup 

Note that when I mount using -v that backup directory is automatically created.

I hope this is useful to someone someday. :)

@mulg0r 2018-04-26 09:11:41

If you use docker-compose, volumes-from is deprecated in version 3 and after.

@ToolmakerSteve 2019-04-03 11:20:07

To add to mulg0r's comment, see stackoverflow.com/a/45495380/199364 - in v.3, you place a volumes command at root of config.yml, for volumes to be accessible by multiple containers.

@BobMcGee 2015-09-13 04:39:01

As a more general solution, there's a CloudBees plugin for Jenkins to build inside a Docker container. You can select an image to use from a Docker registry or define a Dockerfile to build and use.

It'll mount the workspace into the container as a volume (with appropriate user), set it as your working directory, do whatever commands you request (inside the container). You can also use the docker-workflow plugin (if you prefer code over UI) to do this, with the image.inside() {} command.

Basically all of this, baked into your CI/CD server and then some.

@Dimchansky 2015-03-30 15:28:52

Mount a volume, copy the artifacts, adjust owner id and group id:

mkdir artifacts
docker run -i --rm -v ${PWD}/artifacts:/mnt/artifacts centos:6 /bin/bash << COMMANDS
ls -la > /mnt/artifacts/ls.txt
echo Changing owner from \$(id -u):\$(id -g) to $(id -u):$(id -u)
chown -R $(id -u):$(id -u) /mnt/artifacts
COMMANDS

Related Questions

Sponsored Content

24 Answered Questions

[SOLVED] How do I get into a Docker container's shell?

39 Answered Questions

[SOLVED] Copying files from host to Docker container

16 Answered Questions

[SOLVED] How do I copy a file in Python?

50 Answered Questions

[SOLVED] How to get a Docker container's IP address from the host

  • 2013-06-17 22:10:22
  • Murali Allada
  • 1149112 View
  • 1284 Score
  • 50 Answer
  • Tags:   docker

19 Answered Questions

[SOLVED] How is Docker different from a virtual machine?

57 Answered Questions

[SOLVED] How to remove old Docker containers

  • 2013-06-21 13:41:42
  • qkrijger
  • 699224 View
  • 1183 Score
  • 57 Answer
  • Tags:   docker

11 Answered Questions

[SOLVED] What is the difference between the 'COPY' and 'ADD' commands in a Dockerfile?

  • 2014-07-25 14:31:20
  • Steve
  • 630948 View
  • 2050 Score
  • 11 Answer
  • Tags:   docker dockerfile

14 Answered Questions

[SOLVED] How to copy Docker images from one host to another without using a repository

  • 2014-05-29 13:57:18
  • Larry Cai
  • 535897 View
  • 1233 Score
  • 14 Answer
  • Tags:   docker

14 Answered Questions

[SOLVED] How to deal with persistent storage (e.g. databases) in Docker

Sponsored Content