Docker in LXC container on TOS6

I have for a long time run docker in a LXC container on TOS5. It worked well, used instructions from here Docker on Turris Omnia - #44 by ackstorm23

Now after upgrading to TOS6 my docker-lxc fails to start with

lxc-start: ha: cgroups/cgfsng.c: cgfsng_mount: 1810 No such file or directory - Failed to create cgroup at_mnt 26()
lxc-start: ha: conf.c: lxc_mount_auto_mounts: 845 No such file or directory - Failed to mount "/sys/fs/cgroup"

I’m guessing it’s related to cgroups and possibly cgroups v1 and v2. But I don’t know how to fix it. Any ideas?

1 Like

On TOS 6.0 you can run docker natively. Maybe you want to consider that then nesting containers.
But for now try to replaceing cgroups with cgroups2 in your container config. I think some other user reported something similar related to cgroups and it worked.

For sure, I will look into running docker natively. But first I would like to get my current containers up and running. I’ll look into the config and see if I can switch to cgroups2. It’s not obvious to me at a first glance :grinning:

This is the output of lxc-checkconfig:

LXC version 4.0.10
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd

Cgroup v2 mount points:
/sys/fs/cgroup

Cgroup v1 freezer controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: missing
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: missing
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

In my case I just needed to update a raw.lxc with these changes:

  • lxc.cgroup.device.allow to lxc.cgroup2.device.allow
  • Remove line lxc.mount.auto = cgroup:rw:force

This is my working config:

raw.lxc: |-
lxc.cgroup2.devices.allow = a
security.nesting: "true"
security.privileged: "true"
lxc.cap.drop =

And now I can start exploring how to move to native Docker in OpenWRT instead of nested Docker in LXC :slight_smile:

3 Likes

Didn’t get it to work, running native Docker now…

Hello,

The LXC starts correctly with the config mentioned, but inside the LXC, docker daemon won’t start because it says that there isn’t cgroup mounted. See my logs from /var/log/docker.log

time=“2022-10-20T15:56:40.682280836Z” level=info msg=“ClientConn switching balancer to “pick_first”” module=grpc
time=“2022-10-20T15:56:40.685021406Z” level=info msg="[graphdriver] using prior storage driver: btrfs"
time=“2022-10-20T15:56:40.965405322Z” level=warning msg=“Your kernel does not support cgroup memory limit”
time=“2022-10-20T15:56:40.965496560Z” level=warning msg=“Unable to find cpu cgroup in mounts”
time=“2022-10-20T15:56:40.965541274Z” level=warning msg=“Unable to find blkio cgroup in mounts”
time=“2022-10-20T15:56:40.965579834Z” level=warning msg=“Unable to find cpuset cgroup in mounts”
time=“2022-10-20T15:56:40.965616860Z” level=warning msg=“Unable to find pids cgroup in mounts”
time=“2022-10-20T15:56:40.966831101Z” level=info msg=“stopping healthcheck following graceful shutdown” module=libcontainerd
time=“2022-10-20T15:56:40.967322690Z” level=info msg=“stopping event stream following graceful shutdown” error=“context canceled” module=libcontainerd namespace=plugins.moby
failed to start daemon: Devices cgroup isn’t mounted

any idea?

Finally I moved all my dockers to the native system in TOS6.