Docker Discuss Using Docker on CentOS 7 ? Choose EXT4 over XFS

eva2000 · Aug 6, 2017

Another gotcha moment I came across playing with Docker on CentOS 7 in that new Docker versions will need d_type support enabled in filesystem. Docker requires CentOS 7 to use and CentOS /RHEL 7 by default create XFS filesystems with d_type=0 disabled instead of d_type=1. CentOS 7 though creates EXT4 filesystems with d_type=1 enabled by default. See explanation at What is d_type and why Docker overlayfs need it - Linuxer Pro.

Worse thing is you can't fix it on existing CentOS 7 created XFS partitions. You need to recreate the filesystem if you setup CentOS 7 filesystem with XFS partitions !

What is d_type and why Docker overlayfs need it

In my previous post I’ve mentioned a strange problem that occurs on Discourse running in Docker. Today I’m going to explain this further as this problem could potentially impact any Docker setup uses overlayfs storage driver. Practically, CentOS 7 with all default setup during installation is 100% affected. Docker on Ubuntu uses AUFS so is not affected.

What is d_type
d_type is the term used in Linux kernel that stands for “directory entry type”. Directory entry is a data structure that Linux kernel used to describe the information of a directory on the filesystem. d_type is a field in that data structure which represents the type of an “file” the directory entry points to. It could a directory, a regular file, some special file such as a pipe, a char device, a socket file etc.

d_type information was added to Linux kernel version 2.6.4. Since then Linux filesystems started to implement it over time. However still some filesystem don’t implement yet, some implement it in a optional way, i.e. it could be enabled or disabled depends on how the user creates the filesystem.
Click to expand...

Docker info command error when d_type=0 and status on CentOS 7.3 virtualbox install
Code (Text):
docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.06.0-ce
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: false
Code (Text):
WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
        Reformat the filesystem with ftype=1 to enable d_type support.
        Running without d_type support will not be supported in future releases.
Docker info command when d_type=1 enabled on CentOS 7.3 system with EXT4 filesystem
Code (Text):
docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 4
Server Version: 17.06.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true

eva2000 · Aug 7, 2017

Guess one solution is if you have free unallocated disk space, you can create a filesystem with d_type=1 enabled and change Docker's default data directory path from /var/lib/docker to point to the new filesystem

How to change docker installation directory in centos 7 with docker 1.13 version

Control and configure Docker with systemd

i.e. setting up custom settings in /etc/docker/daemon.json
Runtime directory and storage driver
You may want to control the disk space used for Docker images, containers and volumes by moving it to a separate partition.

To accomplish this, set the following flags in the daemon.json file:
Code (Text):
{
   "graph": "/mnt/docker-data",
   "storage-driver": "overlay"
}
Click to expand...
default Docker root directory where data is located
Code (Text):
docker info 2>&1 | grep Root
Docker Root Dir: /var/lib/docker
from Fix strange chown error during Discourse bootstrap or rebuild - Linuxer Pro

Since Docker puts its files under /var/lib/docker directory, you only need to make sure d_type is supported for this specific directory. So if you have free space on your disk, you don’t have to touch your whole root filesystem. Just allocate some space, create a new filesystem with correct parameter, then mount it under /var/lib/docker and it’s done.

As regards to Discourse, this procedure won’t even hurt your Discourse data. Discourse puts all data under /var/discourse/share directory. When you get a new /var/lib/docker directory, only the container definition is gone. You just need to recreate the container with launcher script, then the site will just back to normal.

With that said, backup data before doing any filesystem or disk related operation is still a good practice!
Click to expand...

eva2000 · Aug 7, 2017

Mentioned in RHEL 7.2 preview docs for overlayfs filesystem though Chapter 21. File Systems

OverlayFS
OverlayFS is a type of union file system. It allows the user to overlay one file system on top of another. Changes are recorded in the upper file system, while the lower file system remains unmodified. This allows multiple users to share a file-system image, such as a container or a DVD-ROM, where the base image is on read-only media. Refer to the kernel file Documentation/filesystems/overlayfs.txt for additional information.
Click to expand...

Note that XFS file systems must be created with the -n ftype=1 option enabled for use as an overlay. With the rootfs and any file systems created during system installation, set the --mkfsoptions=-n ftype=1 parameters in the Anaconda kickstart. When creating a new file system after the installation, run the # mkfs -t xfs -n ftype=1 /PATH/TO/DEVICEcommand. To determine whether an existing file system is eligible for use as an overlay, run the # xfs_info /PATH/TO/DEVICE | grep ftype command to see if the ftype=1 option is enabled.
Click to expand...

buik · Aug 7, 2017

@eva2000 Already sent a ticket to Centos bug tracker?

eva2000 · Aug 7, 2017

bassie said: ↑

@eva2000 Already sent a ticket to Centos bug tracker?
Click to expand...

nah I haven't only really read the tracker, never submitted to it before heh.

edit: seems similar bug at 0012031: Removing files located in a lowerdir in an overlay fs is erroneous - CentOS Bug Tracker

buik · Aug 7, 2017

eva2000 said: ↑

nah I haven't only really read the tracker, never submitted to it before heh.

edit: seems similar bug at 0012031: Removing files located in a lowerdir in an overlay fs is erroneous - CentOS Bug Tracker
Click to expand...

Always send tickets.
Otherwise it is not clear to the developer.
No news is good news.

eva2000 · Aug 7, 2017

Yeah but could be argued it's in Docker documentation Use the OverlayFS storage driver so if you use Docker ensure XFS filesystem uses ftype=1 or use EXT4 I guess.

Guess it's partly on me not reading the latest version of the documentation heh.

Prerequisites
OverlayFS is supported if you meet the following prerequisites:

Docker CE only. OverlayFS is not supported on Docker EE. See Product compatibility matrix for a list of supported storage drivers for each Docker EE platform.

Version 4.0 or higher of the Linux kernel. If you use an older kernel, you will need to use the overlay driver, which is not recommended.

The following backing filesystems are supported:

ext4 (RHEL 7.1 only)

xfs (RHEL 7.2 and higher), but only with d_type=true enabled. Use xfs_info to verify that the ftype option is set to 1. To format an xfs filesystem correctly, use the flag -n ftype=1.

Changing the storage driver will make any containers you have already created inaccessible on the local system. Use docker save to save containers, and push existing images to Docker Hub or a private repository, so that you not need to re-create them later.

Click to expand...

Good news is from checking, DigitalOcean, Vultr, and Linode all default to using EXT4 filesystem for CentOS 7 instead of CentOS 7 default XFS. Guess it's for resizing downward support. So CentOS 7 on these VPS providers will be fine with Docker and overlayfs usage as ftype=1 is set.

Which means only folks inconvenienced would mainly be non-vps/cloud dedicated server uses who most likely will partition their own drives etc. So should properly partition or leave unallocated space at initial install time to be facilitate creating a XFS ftype=1 enabled filesystem(s) for their Docker usage.

eva2000 · Aug 7, 2017

example of this issue playing out when folks try to use Discourse on CentOS 7 via docker when XFS filesystem created with ftype=0 / d_type disabled

Installation failing on CentOS 7.3

Bootstrap fails on CentOS 7

The solution:

The easy way - just create another fs for /var/lib/docker with proper option if you have free space on disk. There are two options:
a. Create a ext4 fs with default option and it will be fine.
b. If you prefer xfs, create it like this: mkfs.xfs -n ftype=1 /path/to/your/device

Or reinstall OS and during installation, make sure you manually create root fs with proper option. You cannot specify that option in the GUI/TUI installer so have to run the command by hand in the terminal provided by the installer. The command is mkfs -t xfs -n ftype=1 /PATH/TO/DEVICE per RH document.

Or change your current root fs option from the default ftype=0 to ftype=1. Unfortunately there is no online conversion way exists so you can only do a xfsdump/xfsrestore. Basically you need to prepare a new replacement root partition or disk, manually create xfs on it with proper option, then dump original root fs to the new one, and boot from the new root fs finally.

With any of above solutions applied, in docker info output you will see support d_type = true. Then you won't run into the issue.
Click to expand...

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

Docker Discuss Using Docker on CentOS 7 ? Choose EXT4 over XFS

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

buik Well-Known Member

eva2000 Administrator Staff Member

buik Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

Docker Discuss Using Docker on CentOS 7 ? Choose EXT4 over XFS

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

buik Well-Known Member

eva2000 Administrator Staff Member

buik Well-Known Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.