Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
docker_notes:docker [2024-04-01 Mon wk14 11:34] – [Docker CLI] baumkpdocker_notes:docker [2025-11-27 Thu wk48 17:12] (current) – [Portainer agent] baumkp
Line 1: Line 1:
 {{tag>linux docker cli installation network volume iptables nftables portainer}} {{tag>linux docker cli installation network volume iptables nftables portainer}}
-Much of this material was originally sourced from: [[https://github.com/xcad2k/cheat-sheets/blob/main/docker/docker.md|xcad2k cheat-sheets/docker/docker.md]] 
- 
  
 ======Docker====== ======Docker======
Line 10: Line 8:
  
 =====Installation===== =====Installation=====
 +  *[[https://docs.docker.com/engine/install/debian/|Install Docker Engine on Debian]]
 One click installation script: One click installation script:
   *''%%curl -fsSL https://get.docker.com -o get-docker.sh%%''   *''%%curl -fsSL https://get.docker.com -o get-docker.sh%%''
Line 22: Line 20:
  
 Install Docker Engine : [Docker Engine](https://docs.docker.com/engine/install/) Install Docker Engine : [Docker Engine](https://docs.docker.com/engine/install/)
 +
 +====downgrade docker====
 +The upgrade of docker-ce from version 28.5.2 to 29.0.0 seems to have broken something and the environment stopped function correctly, seems to be docker-ce and traefik related, both packages seem to have been fixed about 3 days later.  The immediate solution was to downgrade the docker-ce version back to the previous version that still function correctly and hope they release a fix the newer version upstream in the near future.
 +For debian based systems:
 +  *''sudo apt policy docker-ce''
 +    *<code>docker-ce:
 +Installed: 5:29.0.0-1~debian.13~trixie
 +Candidate: 5:29.0.0-1~debian.13~trixie
 +Version table:
 +*** 5:29.0.0-1~debian.13~trixie 500
 +      500 https://download.docker.com/linux/debian trixie/stable amd64 Packages
 +      100 /var/lib/dpkg/status
 +   5:28.5.2-1~debian.13~trixie 500
 +      500 https://download.docker.com/linux/debian trixie/stable amd64 Packages
 +   5:28.5.1-1~debian.13~trixie 500
 +      500 https://download.docker.com/linux/debian trixie/stable amd64 Packages
 +   5:28.5.0-1~debian.13~trixie 500</code>
 +  *''sudo apt install docker-ce=5:28.5.2-1~debian.13~trixie''
 +
 +This problem keeps on giving.  A few days later I notice that my Portainer instance, that was running in Docker could not see the VM running portainer docker instances, it show the error: "Failed loading environment".  The remote Portainer agents were operating.  The Local Portainer instance runs on a Socket, whereas the agents communicate via IP, perhaps this is related?  In anycase the solution was edit the docker.service as follows:
 +++++sudo systemctl edit docker.service|
 +<code>[Service]
 +Environment=DOCKER_MIN_API_VERSION=1.24</code>
 +And then ''sudo systemctl restart docker.service'' to restart docker to implement this change.
 +++++
 +Presumably this will eventually be resolved up stream and this fix can be removed.
 +
 +Some references to assist with this:
 +  *[[https://www.linuxuprising.com/2019/02/how-to-downgrade-packages-to-specific.html|How To Downgrade Packages To A Specific Version With Apt In Debian]]
 +    *''sudo apt policy <package>'' to find correct package versions available
 +    *''sudo apt install <package>=<version>'' to install the desired available package version
 +  *[[https://www.linuxuprising.com/2018/10/how-to-keep-package-from-updating-in.html|How To Keep A Package From Updating In Debian]]
 +    *''sudo apt-mark hold PACKAGE'' to hold a PACKAGE from upgrading
 +    *''sudo apt-mark showhold'' to show pakges that have been marked as held
 +    *''sudo apt-mark unhold PACKAGE'' to remove a hold mark from PACKAGE
 +  *[[https://www.linuxuprising.com/2018/10/how-to-search-available-packages-from.html|How To Search For Available Packages From Command Line In Debian [APT]]]
 +    *''sudo apt-cache search KEYWORD'' e.g. ''sudo apt-cache search docker-ce''
 +    *''sudo apt search KEYWORD'' e.g. ''sudo apt search docker-ce''
 +====Docker Desktop for Linux====
 +I have preferred to run Docker command as a native Linux applications.  Docker Desktop originally was created to allow operating systems other than Linux to run Docker by creating a virtual Linux machine to operate them within.  Linux does not need this as the various docker programs run natively. A version of Docker Desktop was created for Linux that looks like it has some additional user interface features, but to date I have not wanted to setup an additional VM for this purpose and am happy to continues to use the Linux KVM VM solution.
  
 =====Uninstall===== =====Uninstall=====
 Both the install methods actually use the standard package manager to install docker. Both the install methods actually use the standard package manager to install docker.
-  -''dpkg -l|grep docker'' to check the docker packages actually installed+  -''dpkg -l|grep docker'' to check the docker packages actually installed and **which docker packages and versions are installed**
   -''sudo apt remove docker-ce''   -''sudo apt remove docker-ce''
   -Check the the ''var/lib/docker'' directory, ''sudo du -d 1 -h var/lib/docker''   -Check the the ''var/lib/docker'' directory, ''sudo du -d 1 -h var/lib/docker''
Line 75: Line 113:
 ++++ ++++
 ++++nft list table ip nat| ++++nft list table ip nat|
-<code># Warning: table ip nat is managed by iptables-nft, do not touch!+<code bash nft.conf># Warning: table ip nat is managed by iptables-nft, do not touch!
 table ip nat { table ip nat {
  chain DOCKER_OUTPUT {  chain DOCKER_OUTPUT {
Line 120: Line 158:
  
 =====Portainer agent===== =====Portainer agent=====
 +Running Portainer agent from docker is a tedious.  I decided to make it operate using compose.
 +++++docker agent docker compose,  docker-compose.yml|
 +<code yaml>name: portainer
 +services:
 +
 +  portainer-agent:
 +    container_name: portainer-agent
 +    image: portainer/agent
 +    ports:
 +      - "9001:9001" 
 +    volumes:
 +      # Mount the host's Docker socket into the container
 +      - /var/run/docker.sock:/var/run/docker.sock
 +      # Mount the host's Docker volumes into the container
 +      - /var/lib/docker/volumes:/var/lib/docker/volumes
 +    deploy:
 +      resources:
 +        limits:
 +          cpus: '0.5'
 +          memory: 1024M
 +      restart_policy:
 +        condition: unless-stopped
 +        delay: 5s
 +        window: 120s</code>
 +++++
 +
 +++++run docker agent from docker|
 Portainer agent allows a remote docker machine to be seen else were via the network.  Default port seems to be 9001. Portainer agent allows a remote docker machine to be seen else were via the network.  Default port seems to be 9001.
   * First stop the agent container: ''%%docker stop portainer_agent%%''   * First stop the agent container: ''%%docker stop portainer_agent%%''
   * Then remove the agent container: ''%%docker rm portainer_agent%%''   * Then remove the agent container: ''%%docker rm portainer_agent%%''
   * Then pull the latest portainer/agent: ''%%docker pull portainer/agent%%'', default is latest if version is not specified.   * Then pull the latest portainer/agent: ''%%docker pull portainer/agent%%'', default is latest if version is not specified.
-<code yaml>docker run -d   -p 9001:9001   --name portainer_agent   --restart=always \ +<code yaml>docker run -d   -p 9001:9001   --name portainer_agent   --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/volumes:/var/lib/docker/volumes portainer/agent</code> 
--v /var/run/docker.sock:/var/run/docker.sock \ +++++ 
--v /var/lib/docker/volumes:/var/lib/docker/volumes \ +It would seem remote agents by default do not show out of date images, can be toggled on/off under ''Host > Setup "Show image up to date indicators for Stacks, Services and Containers"'' 
-portainer/agent</code>+
  
-=====Build Images===== 
  
 =====Docker CLI===== =====Docker CLI=====
 +Much of this material was originally sourced from: [[https://github.com/ChristianLempa/cheat-sheets/blob/main/docker/docker.md|cheat-sheets/docker/docker.md]]
  
 **Run Containers** **Run Containers**
Line 236: Line 301:
 |''docker volume rm VOLUME'' | Destroy a volume| |''docker volume rm VOLUME'' | Destroy a volume|
 |''%%docker volume ls --filter="dangling=true%%"'' | List all dangling volumes (not referenced by any container)| |''%%docker volume ls --filter="dangling=true%%"'' | List all dangling volumes (not referenced by any container)|
-|''docker volume prune'' | Delete all volumes (not referenced by any container)|+|''docker volume prune'' | Delete all volumes not referenced by any container|
  
 +**Network:**
 +|<50em>|
 +^COMMAND ^ DESCRIPTION^
 +|''docker network ls'' | List all volumes|
 +|''docker network create NETWORK'' | Create a NETWORK|
 +|''docker network inspect NETWORK'' | Show information (json formatted)|
 +|''docker network rm NETWORK'' | Destroy a NETWORK|
 +|''%%docker network ls --filter="dangling=true%%"'' | List all dangling networks (not referenced by any container)|
 +|''docker network prune'' | Delete all networks volumes not referenced by any container|
 +
 +=====DOCKER DNS=====
 +Docker has an internal DNS for each discrete type network.  This allows docker containers to be referenced by container name and compose name and allows repeatable inter-container referencing by name, so dynamic  container internal IP addressing is not a reference issue.  Container names are automatically assigned or can be specifically assigned with ''--name CONTAINER_NAME'' in Docker, and ''container_name: CONTAINER_NAME'' in Docker compose file.
 ====docker ps command==== ====docker ps command====
 The ''docker ps'' command output is long and often difficult to read on the terminal for this reason. The ''docker ps'' command output is long and often difficult to read on the terminal for this reason.
-''%%docker ps --format 'table {{ .ID }}\t{{.Image}}\t{{ .Names }}'%%'' +  *''%%docker ps --format 'table {{ .ID }}\t{{.Image}}\t{{ .Names }}'%%'' 
-CONTAINER ID   IMAGE                                           NAMES +<code>CONTAINER ID   IMAGE                    NAMES</code> 
-''%%docker ps -s --format 'table {{ .ID }}\t{{.Status}}\t{{.Image}}\t{{ .Names }}'%%'' +  *''%%docker ps -s --format 'table {{ .ID }}\t{{.Status}}\t{{.Image}}\t{{ .Names }}'%%'' 
-CONTAINER ID   STATUS                  IMAGE                                           NAMES +<code>CONTAINER ID   STATUS                  IMAGE                 NAMES</code> 
-''%%docker ps -s --format 'table {{ .ID }}\t{{.Status}}\t{{.Image}}\t{{ .Names }}\t{{.Size}}'%%'' +  *''%%docker ps -s --format 'table {{ .ID }}\t{{.Status}}\t{{.Image}}\t{{ .Names }}\t{{.Size}}'%%'' 
-CONTAINER ID   STATUS                  IMAGE                                           NAMES               SIZE +<code>CONTAINER ID   STATUS                  IMAGE                 NAMES               SIZE</code>
  
 +===reference===
 +  *[[https://devdojo.com/bobbyiliev/how-to-change-the-docker-ps-output-format|How to change the docker ps output format]]
 +  *[[https://dev.to/cicube/docker-cheat-sheet-most-useful-commands-ghl|Docker Cheat Sheet - Most Useful Commands]]
 +  *Docker Docs
 +    *[[https://docs.docker.com/reference/cli/docker/container/ls/|docker container ls]]
 +    *[[https://docs.docker.com/storage/storagedriver/#container-size-on-disk|Container size on disk]]
 ====Backup a container==== ====Backup a container====
 Backup docker data from inside container volumes and package it in a tarball archive.\\ Backup docker data from inside container volumes and package it in a tarball archive.\\
Line 271: Line 353:
     * ''docker network create network_named''     * ''docker network create network_named''
   - Host (Appears on the host machine as if installed there, no separate network.)   - Host (Appears on the host machine as if installed there, no separate network.)
-  - MACVLAN +    -If you use the host network mode for a container, that container's network stack isn't isolated from the Docker host (the container shares the host's networking namespace), and the container doesn't get its own IP-address allocated.   
 +  - MACVLAN 
 +    -The macvlan network assigns a unique MAC address to each container, making it appear to be a physical device on your network, just like a traditional virtual machine. The Docker daemon then routes the traffic to containers on the basis of their MAC address. It also allows you to assign an IP address from the same subnet in which the Docker host resides. This avoids the use of the host network, there is no NAT overhead, and you won't run into network performance issues.  
     - MACVLAN (without subVLAN) this create a new ip address on the host machine     - MACVLAN (without subVLAN) this create a new ip address on the host machine
-      * <code bash>docker network create -d macvlan \+      * <code bash>docker network create \ 
 +-d macvlan \
 --subnet 192.168.1.0/24 \ --subnet 192.168.1.0/24 \
 --gateway 192.168.1.1 \ --gateway 192.168.1.1 \
--o parent=br0 \ +-o parent=br0 network_named</code>
-network_named</code>+
       * No host DHCP access so need to specify ip address when creating container (docker cli ''%%--ip 192.168.1.14%%''). If not specified docker DHCP will assign and could cause clash with host.       * No host DHCP access so need to specify ip address when creating container (docker cli ''%%--ip 192.168.1.14%%''). If not specified docker DHCP will assign and could cause clash with host.
       * May be problem with multiple MACs on common switch port.  Need to set promiscuous mode on network, e.g. ''sudo ip link set br0 promisc on''.       * May be problem with multiple MACs on common switch port.  Need to set promiscuous mode on network, e.g. ''sudo ip link set br0 promisc on''.
Line 283: Line 367:
   - IPVLAN    - IPVLAN 
     - IPVLAN on host subnet, this create a new ip address on the host machine, but not with new MAC number, it uses the host MAC number     - IPVLAN on host subnet, this create a new ip address on the host machine, but not with new MAC number, it uses the host MAC number
-      * <code bash>docker network create -d ipvlan \+      * <code bash>docker network create \ 
 +-d ipvlan \
 --subnet 192.168.1.0/24 \ --subnet 192.168.1.0/24 \
 --gateway 192.168.1.1 \ --gateway 192.168.1.1 \
--o parent=br0 \ +-o parent=br0 network_named</code>
-network_named</code>+
       * No host DHCP access so need to specify ip address when creating container (docker cli ''%%--ip 192.168.1.14%%''). If not specified docker DHCP will assign and could cause clash with host.       * No host DHCP access so need to specify ip address when creating container (docker cli ''%%--ip 192.168.1.14%%''). If not specified docker DHCP will assign and could cause clash with host.
       * May be problem with shared MAC with multiple IP address, but less likely than MACVLAN.       * May be problem with shared MAC with multiple IP address, but less likely than MACVLAN.
     - IPVLAN on separate subnet using the host machine as gateway, but not with new MAC number, it uses the host MAC number     - IPVLAN on separate subnet using the host machine as gateway, but not with new MAC number, it uses the host MAC number
-      * <code bash>docker network create -d ipvlan \+      * <code bash>docker network create \ 
 +-d ipvlan \
 --subnet 192.168.1.0/24 \ --subnet 192.168.1.0/24 \
 -o parent=br0 -o ipvlan_mode=l3 \ -o parent=br0 -o ipvlan_mode=l3 \
---subnet 192.168.10.0/24 \ +--subnet 192.168.10.0/24 network_named</code>
-network_named</code>+
       * No host DHCP access so need to specify ip address when creating container (docker cli ''%%--ip 192.168.1.14%%''). If not specified docker DHCP will assign and could cause clash with host.       * No host DHCP access so need to specify ip address when creating container (docker cli ''%%--ip 192.168.1.14%%''). If not specified docker DHCP will assign and could cause clash with host.
       * May be problem with shared MAC with multiple IP address, but less likely than MACVLAN.       * May be problem with shared MAC with multiple IP address, but less likely than MACVLAN.
Line 301: Line 385:
   - None network - no assigned network, container has no external network connectivity   - None network - no assigned network, container has no external network connectivity
  
-====network troubleshooting====+====References==== 
 +  *[[https://www.aidenwebb.com/posts/dockers-seven-network-types-and-when-to-use-them/|Dockers seven network types and when to use them]] 
 +  *[[https://dev.to/wallacefreitas/docker-networking-a-comprehensive-guide-3d5j|Docker Networking: A Comprehensive Guide]] 
 +  *[[https://dev.to/abhay_yt_52a8e72b213be229/unlocking-advanced-docker-networking-macvlan-vs-ipvlan-38o4|Unlocking Advanced Docker Networking: Macvlan vs. Ipvlan]] 
 +  *[[https://ipwithease.com/macvlan-vs-ipvlan-understand-the-difference/|MacVLAN vs IPvlan: Understand the difference]] 
 +  *[[https://medium.com/@dyavanapellisujal7/docker-macvlan-and-ipvlan-explained-advanced-networking-guide-b3ba20bc22e4|Docker MacVLAN and IPVLAN Explained: Advanced Networking Guide]] 
 +  *[[https://4sysops.com/archives/macvlan-network-driver-assign-mac-address-to-docker-containers/|Macvlan network driver: Assign MAC address to Docker containers]] 
 +  *[[https://4sysops.com/archives/configuring-ipvlan-networking-in-docker/|Configuring IPvlan networking in Docker]] 
 +=====network troubleshooting=====
 A lot of containers are setup to be small and hence do not include many, if any of the tools required to diagnose problems.  A small docker image ''netshoot'' includes the most common networking tools and when attached to the same docker network can be used to diagnose the network and containers networks thereon. A lot of containers are setup to be small and hence do not include many, if any of the tools required to diagnose problems.  A small docker image ''netshoot'' includes the most common networking tools and when attached to the same docker network can be used to diagnose the network and containers networks thereon.
   *''%%docker run --rm --name netshoot --network proxy -it nicolaka/netshoot /bin/bash%%''   *''%%docker run --rm --name netshoot --network proxy -it nicolaka/netshoot /bin/bash%%''
  
-====Troubleshooting====+ 
 +====Network Troubleshooting====
   *[[https://github.com/nicolaka/netshoot|netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container]] ''%%docker run --name netshoot --rm -it nicolaka/netshoot /bin/bash%%''   *[[https://github.com/nicolaka/netshoot|netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container]] ''%%docker run --name netshoot --rm -it nicolaka/netshoot /bin/bash%%''
   *[[https://gist.github.com/bastman/5b57ddb3c11942094f8d0a97d461b430|Docker - How to cleanup (unused) resources]]   *[[https://gist.github.com/bastman/5b57ddb3c11942094f8d0a97d461b430|Docker - How to cleanup (unused) resources]]
 +
 +=====Docker Cleanup=====
 +Docker can take a lot of disk space.  While I store my docker containers and data on a separate drive with a lot of space, docker is run on a VM that has about 55GB space, which is usually enough for Linux.  
 +Docker stores the container images and other operating information in ''/var/lib/docker'', with a lot of information in ''/var/lib/docker/overlay2'', the command ''sudo du -d 1 -h /var/lib/docker'' lists the sub-directories and used space.   See [[https://www.virtualizationhowto.com/2023/11/docker-overlay2-cleanup-5-ways-to-reclaim-disk-space/|Docker Overlay2 Cleanup: 5 Ways to Reclaim Disk Space]], in particular the following commands:
 +  *''docker system prune'', which will remove stopped containers, networks not associated with any container, and dangling images, the ''-a'' flag will also remove all unused images.  The ''-f'' flag is to force removal.
 +  *''docker image prune''  removes older images.
 +
  
 <- docker_notes:vm-container|Back ^ docker_notes:index|Start page ^ docker_notes:docker-compose|Next -> <- docker_notes:vm-container|Back ^ docker_notes:index|Start page ^ docker_notes:docker-compose|Next ->