KVM versus Proxmox

Much of this material was originally sourced from: xcad2k cheat-sheets/docker/docker.md

I originally started using Linux KVM based VM, with QEMU and Libvirt on Ubuntu bare metal, circa 2014, this before was I was aware that Proxmox existed. Around 2020 I moved to Debian as my preferred bare metal distribution for server and desktop, I stopped using Windows as my main desktop around this time. When I started playing around with Docker to create my own container images I preferred use of the Alpine distribution and where necessary using the S6-rc init system. I avoid the use of Ubuntu now as I just find some of their practices unpalatable, e.g. forced use of Snaps, requiring registration for latest package updates. Also their base server and desktop distributions came across as bloated around the time I stopped using them as my main distribution. I see no point reengaging with Ubuntu at this time, as many of the dissatisfaction that made me move are still in place, perhaps even worse now.

Interestingly as I understand it Proxmox uses Debian and Linux KVM VM, however they also provide a lot of additional functionally, such as nice web interface, nice VM backup, etc.

I use nftables as my main edge router / firewall on bare metal, I could probably run this on a VM within Proxmox. My current router is an Intel N3700, which in 2023 is becoming slow and resource limited. I also run a VM with Docker containers for a backup Bind9 DNS and backup Kea DHCP. As my upstream internet speed is about 65Mb/s down and 17Mb/s up this router is still suitable for purpose. I expect when internet connectivity speed goes above 250Mb/s the router capacity may become limiting. (I am currently eyeing an Intel i5-1335U as a possible replacement, this is much fast overall and should easily handle multi Gb/s internet traffic, as well as more complex resource intensive Docker instances.)

My main home server is based upon an Intel Atom C3750 server, this is still currently adequately meeting my needs. I have upgraded with a 2.5Gb/s PCIe card. I have not been able to find a good replacement for this machine at this time. It was designed as a server, again an i5-1335U is in many ways superior, CPU cores and threads, CPU and memory speed and bandwidth, however memory is not ECC and memory is limited to 64GB, neither of which is probably a problem for me, as I am currently only using 32GB. Power consumption is similar. The biggest problem is that I have not been able to date find an i5-1335U motherboard with 4+ SATA ports and PCIE expansion slot, most are laptop boards, router boards or industrial embedded type boards that do not have the functionality that I am after.

I may try Proxmox in the future, there is currently no compelling reason for me to do so at this time.

I use Linux KVM with libvirt, virsh and qemu.

Install standard Debian files. See kvm setup
Add user to libvirt and libvirt-qemu, e.g. sudo usermod -a -G libvirt-qemu baumkp
If you are ssh'ing into the host machine remember to add the ssh key to allow password-less login. e.g. ssh-copy-id 192.168.1.21 from the host machined where 192.168.1.21 is the remote machine. If you do not do this the VM installer can ask for password continuously to the point of making use non-functional.
- The user ssh keys are stored here ~/.ssh/known_hosts. If there is a problem with ssh key have a look here.
  - Check file permission is local user
  - Delete old key in file if necessary
  - Delete whole file if necessary
sudo apt install vim to install vim

Set static ip address and a bridge network (this varies on the install type)

For networkd:

sudo apt install bridge-utils Docker will need a bridge network connection

/etc/network/interfaces

source /etc/network/interfaces.d/*
 
# The loopback network interface
auto lo
iface lo inet loopback
 
# The primary network interface
#allow-hotplug enp1s0
#iface enp1s0 inet dhcp
 
auto br0
iface br0 inet static
  bridge_ports enp1s0
  address 192.168.1.2/24
  gateway 192.168.1.1         #Do not use on router
#  dns-nameservers 192.168.1.1
  bridge_stp off  # disable Spanning Tree Protocol

How to use bridged networking with libvirt and KVM

Docker is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called _containers_. The service has both free and premium tiers. The software that hosts the containers is called Docker Engine.

Project Homepage: [Home - Docker](https://www.docker.com/)
Documentation: [Docker Documentation | Docker Documentation](https://docs.docker.com/)
Docker hub: [Docker hub](https://hub.docker.com)

One click installation script:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

Note for Debian testing just use sudo apt install docker, as testing packages the latest docker and the docker repo does not seem to have testing.

Run docker as non root user:

sudo groupadd docker
sudo usermod -aG docker $USER

Install Docker Engine : [Docker Engine](https://docs.docker.com/engine/install/)

Both the install methods actually use the standard package manager to install docker.

dpkg -l|grep docker to check the docker packages actually installed
sudo apt remove docker-ce
Check the the var/lib/docker directory, sudo du -d 1 -h var/lib/docker
rm -rf /var/lib/docker to remove the docker directory. I try not to store any important files in docker, so this should be low risk for me.

Uninstall docker version installed via script

Docker applies iptables on the host machine, see Docker and iptables.

A Docker host that is acting or running as a router, e.g. using Traefik, the following may be needed to allow operation.

For some more details, tldr;

This may affect other services operating on the host machine. The default FORWARD chain is set to DROP. On my existing home web server this effectively stopped the KVM bridge network from operating, so the virtual machines could not communicate outside the local host IP address with other LAN addresses, including the router / gateway.

Linux forwarding is required for Docker and can be checked with sudo sysctl net.ipv4.ip_forward.

It would also adversely affect my Linux router should I attempt to load docker on this machine. To make matters worse my Linux router is based on nftables which may not operate well with iptables. I will cross that hurdle should I get there. Interestingly, it would seem that iptables operates on/as nftables in the Debian 11.

As perDocker and iptables. Docker on a router the “simple” solution is to add FORWARD ACCEPT rules to the DOCKER-USER chain as required, e.g.: iptables -I DOCKER-USER -i src_if -o dst_if -j ACCEPT. For the machine in question this became sudo iptables -I DOCKER-USER -i br0 -o br0 -j ACCEPT, where br0 was the bridge interface for the VMs and also LAN access. The next problem was how to make this occur automagically after a reboot. I added a new systemd.service for this: sudo vim /etc/systemd/system/iptables-docker.service:

[Unit]
Description=IPTables Docker Rules
After=docker.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/iptables -I DOCKER-USER -i br0 -o br0 -j ACCEPT

[Install]
WantedBy=multi-user.target

As I was just adding a single rule to one chain I simply ran the iptables command as a oneshot service after the docker.service. If I needed a more complex iptable configuration I could have created a file called iptables-docker.rules in /etc/iptables/rules.v4 and then used command iptables-restore -nv < /etc/iptables/rules.v4 or some such in the docker.service.

Some hand commands:

sudo iptables -S to see the current iptable rules
sudo nft list ruleset lists as nftable rules
whereis iptables to find the full path for the iptables command.
systemctl daemon-reload to reload the system service file if modified
sudo systemctl enable iptables-docker.service to enable the service to function automatically, e.g. at startup, on multi-user-target

Portainer is a handy web view into the Docker system. It can be loaded on the host as a Docker container as per this docker documentation Install Portainer with Docker on Linux! A lot of the official Portainer web points to the business version. They do allow 5 free business installations with registration. The portainer-ee image is the business image on Docker hub, portainer-ce is the community addition. When downgrading from the business edition to the community edition the database on the volume storage needs to be downgraded or portainer-ce will not start.

The community addition command is:

docker volume create portainer_data
docker run -d -p 9443:9443 --name=portainer --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    portainer/portainer-ce:latest

Notes:

The port 8000 (-p 8000:8000) is not normally required on community edition. Further to this Kea dhcp control agent uses port 8000 as default.
Some browsers give strange message if http access is requested and then require exception for dodgy https certificate.

Portainer agent allows a remote docker machine to be seen else were via the network. Default port seems to be 9001.

First stop the agent container: docker stop portainer_agent
Then remove the agent container: docker rm portainer_agent
Then pull the latest portainer/agent: docker pull portainer/agent, default is latest if version is not specified.

docker run -d   -p 9001:9001   --name portainer_agent   --restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/docker/volumes:/var/lib/docker/volumes \
portainer/agent

Run Containers

COMMAND	DESCRIPTION
`docker run IMAGE`	Start a new container
`docker run --name CONTAINER IMAGE`	Start a new container and set a name
`docker run -p HOSTPORT:CONTAINERPORT IMAGE`	Start a new container with mapped ports
`docker run -P IMAGE`	Start a new container and map all ports
`docker run -d IMAGE`	Start a new container in background
`docker run -v HOSTDIR:TARGETDIR IMAGE`	Map local directory and Launch

Container Management:

COMMAND	DESCRIPTION
`docker ps`	Show a list of all running containers
`docker ps -a`	Show a list of all containers
`docker create IMAGE`	Create a new container
`docker start CONTAINER`	Start a container
`docker stop CONTAINER`	Graceful stop a container
`docker kill CONTAINER`	Kill (SIGKILL) a container
`docker restart CONTAINER`	Graceful stop and restart a container
`docker pause CONTAINER`	Suspend a container
`docker unpause CONTAINER`	Resume a container
`docker rename OLD NEW`	Rename a container
`docker rm CONTAINER`	Destroy a container
`docker rm -f CONTAINER`	Destroy a running container
`docker exec -it CONTAINER COMMAND`	Run a command in a container
`docker exec -it CONTAINER bash`	Run bash shell in a container

Container Bulk Management

COMMAND	DESCRIPTION
`docker stop $(docker ps -q)`	To stop all the running containers
`docker stop $(docker ps -a -q)`	To stop all the stopped and running containers
`docker kill $(docker ps -q)`	To kill all the running containers
`docker kill $(docker ps -a -q)`	To kill all the stopped and running containers
`docker restart $(docker ps -q)`	To restart all running containers
`docker restart $(docker ps -a -q)`	To restart all the stopped and running containers
`docker rm $(docker ps -q)`	To destroy all running containers
`docker rm $(docker ps -a -q)`	To destroy all the stopped and running containers
`docker pause $(docker ps -q)`	To pause all running containers
`docker pause $(docker ps -a -q)`	To pause all the stopped and running containers
`docker start $(docker ps -q)`	To start all running containers
`docker start $(docker ps -a -q)`	To start all the stopped and running containers
`docker rm -vf $(docker ps -a -q)`	To delete all containers including its volumes use
`docker rmi -f $(docker images -a -q)`	To delete all the images
`docker system prune`	To delete all dangling and unused images, containers, cache and volumes
`docker system prune -a`	To delete all used and unused images
`docker system prune --volumes`	To delete all docker volumes

Inspect / Troubleshoot Containers:

COMMAND	DESCRIPTION
`docker ps`	List running containers
`docker ps -a`	List all containers, including stopped
`docker logs CONTAINER`	Show a container output
`docker logs -f CONTAINER`	Follow a container output
`docker top CONTAINER`	List the processes running in a container
`docker diff`	Show the differences with the image (modified files)
`docker inspect`	Show information of a container (json formatted)
`docker stats`	Show stats
`docker port CONTAINER`	Show mapped port of a container

Run Commands:

COMMAND	DESCRIPTION
`docker attach CONTAINER`	Attach to a container
`docker cp CONTAINER:PATH HOSTPATH`	Copy files from the container
`docker cp HOSTPATH CONTAINER:PATH`	Copy files into the container
`docker export CONTAINER`	Export the content of the container (tar archive)
`docker exec CONTAINER`	Run a command inside a container
`docker exec -it CONTAINER /bin/bash`	Open an interactive shell inside a container (there is no bash in some images, use /bin/sh)
`docker wait CONTAINER`	Wait until the container terminates and return the exit code

Images:

COMMAND	DESCRIPTION
`docker images`	List all local images
`docker history IMAGE`	Show the image history
`docker inspect IMAGE`	Show information (json formatted)
`docker tag IMAGE TAG`	Tag an image
`docker commit CONTAINER IMAGE`	Create an image (from a container)
`docker import URL`	Create an image (from a tarball)
`docker rmi IMAGE`	Delete images
`docker pull REPO:[TAG]`	pull an image/repo from a registry
`docker push REPO:[TAG]`	push and image/repo to a registry
`docker search TEXT`	Search an image on the official registry
`docker login`	Login to a registry
`docker logout`	Logout from a registry
`docker save REPO:[TAG]`	Export an image/repo as a tarball
`docker load`	Load images from a tarball
`docker build DIRECTORY`	Build an image from a docker file

Volumes:

COMMAND	DESCRIPTION
`docker volume ls`	List all volumes
`docker volume create VOLUME`	Create a volume
`docker volume inspect VOLUME`	Show information (json formatted)
`docker volume rm VOLUME`	Destroy a volume
`docker volume ls --filter="dangling=true“`	List all dangling volumes (not referenced by any container)
`docker volume prune`	Delete all volumes (not referenced by any container)

Backup docker data from inside container volumes and package it in a tarball archive.
docker run --rm --volumes-from CONTAINER -v $(pwd):/backup busybox tar cvfz /backup/backup.tar CONTAINERPATH

Restore the volume with a tarball archive.
docker run --rm --volumes-from CONTAINER -v $(pwd):/backup busybox sh -c "cd CONTAINERPATH && tar xvf /backup/backup.tar --strip 1"

Docker volume types:

Host Volumes: -v /path/on/host:/path/in/container
Anonymous Volumes: -v /path/in/container
Named Volumes: -v name:/path/in/container

Docker controlled volumes are stored at /var/lib/docker/volumes

Type of networks on Docker:

Default Bridge (The default separate isolated bridge network.)
- This is the default network used if nothing else is defined
User defined bridge (DNS name resolution on Docker containers, which default bridge does not.)
- docker network create network_named
Host (Appears on the host machine as if installed there, no separate network.)
MACVLAN
1. MACVLAN (without subVLAN) this create a new ip address on the host machine
  - ```
  docker network create -d macvlan \
  --subnet 192.168.1.0/24 \
  --gateway 192.168.1.1 \
  -o parent=br0 \
  network_named
```
- No host DHCP access so need to specify ip address when creating container (docker cli --ip 192.168.1.14). If not specified docker DHCP will assign and could cause clash with host.
- May be problem with multiple MACs on common switch port. Need to set promiscuous mode on network, e.g. sudo ip link set br0 promisc on.
2. MACVLAN with subVLAN - never had to use this yet myself, so do not know much about it, something about trunking on a common link.
IPVLAN
1. IPVLAN on host subnet, this create a new ip address on the host machine, but not with new MAC number, it uses the host MAC number
  - ```
  docker network create -d ipvlan \
  --subnet 192.168.1.0/24 \
  --gateway 192.168.1.1 \
  -o parent=br0 \
  network_named
```
- No host DHCP access so need to specify ip address when creating container (docker cli --ip 192.168.1.14). If not specified docker DHCP will assign and could cause clash with host.
- May be problem with shared MAC with multiple IP address, but less likely than MACVLAN.
2. IPVLAN on separate subnet using the host machine as gateway, but not with new MAC number, it uses the host MAC number
  - ```
  docker network create -d ipvlan \
  --subnet 192.168.1.0/24 \
  -o parent=br0 -o ipvlan_mode=l3 \
  --subnet 192.168.10.0/24 \
  network_named
```
- No host DHCP access so need to specify ip address when creating container (docker cli --ip 192.168.1.14). If not specified docker DHCP will assign and could cause clash with host.
- May be problem with shared MAC with multiple IP address, but less likely than MACVLAN.
Overlay network, an even more obscure network arrangement I know nothing about.
None network - no assigned network, container has no external network connectivity

netshoot: a Docker + Kubernetes network trouble-shooting swiss-army container docker run --name netshoot --rm -it nicolaka/netshoot /bin/bash
Docker - How to cleanup (unused) resources

← Back

Start page

Next →

KVM versus Proxmox

Setup VM

Reference

Docker

Installation

Uninstall

Docker and iptables

Portainer

Portainer agent

Build Images

Docker CLI

Backup a container

Restore container from backup

Volumes

Networks

Troubleshooting

KPTree Wiki