AlmaLinux usertmp_donotdelete on DigitalOcean / Rocky8 causing RO file system on reboot

Interesting issue. How old is this 130.00beta01 installation? You can find out running command

Code (Text):

cminfo versions

If you haven't ran cminfo before, run it by itself once before running above command as some dependencies might need to be installed. The command lists a history of Centmin Mod versions from first install and subsequent updates when you run centmin.sh menu. Did you have same resize problem with CentOS 7 systems or just EL8 specific? What filesystem XFS or EXT4?

Code (Text):

df -hT

output?

The routine that would setup /home/usertmp_donotdelete for secure /tmp directory setup is in 130.00beta01 centmin.sh code at lines 2196-2450.

The /home/usertmp_donotdelete file is created under the following conditions:

1. The system is not an OpenVZ virtualization or LXD container.
2. The system is running CentOS 7 or a non-CentOS 7 system.
3. The total memory of the system falls within specific ranges:
   a. Between 2GB and <8GB
   b. Between 1.1GB and <2GB
4. Based on the available free disk space on /home partition, the size of the /home/usertmp_donotdelete file will be determined.

The original aim in creating the separate /tmp loop device was due to default /tmp being memory based (tmpfs) and if you potentially don't have enough memory to keep /tmp as memory based (tmpfs) filesystem, there's a chance you'd run into issues. So for <8GB memory systems, setup the disk based /tmp drive as a solution. So if you see on Centmin Mod installs with have >8GB of memory, they don't have /tmp setup as disk based using /home/usertmp_donotdelete mounts in /etc/fstab. You can see an example on this 4GB KVM VPS, it will for non-Centmin Mod systems setup /tmp as tmpfs ram disk mount to 1/2 memory size = 2GB for /tmp. With Centmin Mod, it will detect <8GB memory system and setup a non-tmpfs non-ramdisk mount for /tmp that is larger size at 3.7GB so that apps tmp drive usage requirements don't put pressure on memory and are more adequate for /tmp usage.

Code (Text):

df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs          tmpfs     2.0G  8.5M  1.9G   1% /run
tmpfs          tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda2      xfs        80G  7.1G   73G   9% /
/dev/loop0     ext4      3.9G   76K  3.7G   1% /tmp
tmpfs          tmpfs     390M     0  390M   0% /run/user/0

But yes seems that is the expected outcome from what you outlined. The solution is to do a remount after a resize of the disk if you want to keep /etc/fstab tmp directory loop device entry. So resize and then remount. But are you saying normal server reboots (without disk resize) don't work with /etc/fstab /tmpdir loop device entry? Or only whenever you do a disk resize? If it's the latter, the correct way is to always do a remount after a disk resize to ensure /etc/fstab tmp directory loop device entry works and disk isn't in read only mode.

Edit: better solution https://community.centminmod.com/th...ng-ro-file-system-on-reboot.23763/#post-96171

 

For read-only remount, try with

Code (Text):

mount -o remount,rw /

 

try this in read-only state do the following

Code (Text):

mount -o remount,rw /
mount -a

and then try subsequent reboots and see if it's still read only

 

If that doesn't work try this method instead

create /usr/local/bin/mount-loop.sh shell script and give it executable permissions (chmod +x /usr/local/bin/mount-loop.sh)

add to the file

Code (Text):

#!/bin/bash
mount -o loop,rw,noexec,nosuid /home/usertmp_donotdelete /tmp

Remove the /home/usertmp_donotdelete /tmp ext4 loop,rw,noexec,nosuid 0 0 entry from /etc/fstab.

Create /etc/systemd/system/mount-loop.service with following content

Code (Text):

[Unit]
Description=Mount loop device for /tmp

[Service]
ExecStart=/usr/local/bin/mount-loop.sh
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Then enable the mount-loop.service

Code (Text):

systemctl enable mount-loop.service

Then reboot server and check df -hT command to see if /tmp is mounted and disk is not read-only

 

Looks like I need to update Centmin Mod's tmp directory mount routine to use systemd service

 

@Matt can you try this updated /etc/systemd/system/mount-loop.service

Code (Text):

[Unit]
Description=Mount loop device for /tmp
After=local-fs.target
Requires=local-fs.target

[Service]
ExecStart=/usr/local/bin/mount-loop.sh
ExecStop=/usr/bin/umount /tmp
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

then try

Code (Text):

systemctl daemon-reload
systemctl enable mount-loop.service

and then reboot your system and see if that works. This is probably what I will update for 130.00beta01 as it will also allow you to unmount the /tmp loop device too via

Code (Text):

systemctl stop mount-loop.service

edit: works at least on fresh Upcloud 2CPU KVM VPS fresh install of private EL8 Centmin Mod 130.00beta01 with AlmaLinux 8.7

Code (Text):

systemctl status mount-loop
● mount-loop.service - Mount loop device for /tmp
   Loaded: loaded (/etc/systemd/system/mount-loop.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2023-04-25 14:02:37 UTC; 7min ago
  Process: 700 ExecStart=/usr/local/bin/mount-loop.sh (code=exited, status=0/SUCCESS)
 Main PID: 700 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 24704)
   Memory: 0B
   CGroup: /system.slice/mount-loop.service
Apr 25 14:02:37 almalinux-2cpu-4gb-us-nyc1 systemd[1]: Starting Mount loop device for /tmp...
Apr 25 14:02:37 almalinux-2cpu-4gb-us-nyc1 systemd[1]: Started Mount loop device for /tmp.

Code (Text):

df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs          tmpfs     2.0G  8.5M  1.9G   1% /run
tmpfs          tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda2      xfs        80G  7.1G   73G   9% /
/dev/loop0     ext4      3.9G   76K  3.7G   1% /tmp
tmpfs          tmpfs     390M     0  390M   0% /run/user/0