Learn about Centmin Mod LEMP Stack today
Register Now

AlmaLinux usertmp_donotdelete on DigitalOcean / Rocky8 causing RO file system on reboot

Discussion in 'Beta release code' started by Matt, Apr 25, 2023.

  1. Matt

    Matt Well-Known Member

    917
    413
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +664
    Local Time:
    11:25 PM
    1.5.15
    MariaDB 10.2
    Please fill in any relevant information that applies to you:
    • CentOS Version: Rocky8
    • Centmin Mod Version Installed: 130.00beta01
    • Nginx Version Installed: 1.23.4
    • PHP Version Installed: PHP8.0.28
    • MariaDB MySQL Version Installed: NA, running stand alone DB server.
    • When was last time updated Centmin Mod code base ? : Brand New Install and existing install as well.
    • Persistent Config: No

    This has been bugging me for over a week now! On a server I built at the start of the year, we re-sized it using the DO rescale function. On power on, the server came back up with RO file system.

    I tried everything to see why it was on RO mode, booted in recovery, ran xfs_repair / fsck etc, didn't fix it.

    Rebuilt a new VM today, moved everything over, and ran resize option again.....bang, same issue. Back in RO mode.

    Got it back up by mounting / again

    mount -o remount /

    And editing out the /tmp entry

    #/home/usertmp_donotdelete /tmp ext4 loop,rw,noexec,nosuid 0 0

    With this removed, it allows the server to reboot and start correctly.

     
  2. eva2000

    eva2000 Administrator Staff Member

    52,262
    12,005
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,508
    Local Time:
    8:25 AM
    Nginx 1.25.x
    MariaDB 10.x
    Interesting issue. How old is this 130.00beta01 installation? You can find out running command
    Code (Text):
    cminfo versions
    

    If you haven't ran cminfo before, run it by itself once before running above command as some dependencies might need to be installed. The command lists a history of Centmin Mod versions from first install and subsequent updates when you run centmin.sh menu. Did you have same resize problem with CentOS 7 systems or just EL8 specific? What filesystem XFS or EXT4?
    Code (Text):
    df -hT

    output?

    The routine that would setup /home/usertmp_donotdelete for secure /tmp directory setup is in 130.00beta01 centmin.sh code at lines 2196-2450.

    The /home/usertmp_donotdelete file is created under the following conditions:
    1. The system is not an OpenVZ virtualization or LXD container.
    2. The system is running CentOS 7 or a non-CentOS 7 system.
    3. The total memory of the system falls within specific ranges:
      a. Between 2GB and <8GB
      b. Between 1.1GB and <2GB
    4. Based on the available free disk space on /home partition, the size of the /home/usertmp_donotdelete file will be determined.
    The original aim in creating the separate /tmp loop device was due to default /tmp being memory based (tmpfs) and if you potentially don't have enough memory to keep /tmp as memory based (tmpfs) filesystem, there's a chance you'd run into issues. So for <8GB memory systems, setup the disk based /tmp drive as a solution. So if you see on Centmin Mod installs with have >8GB of memory, they don't have /tmp setup as disk based using /home/usertmp_donotdelete mounts in /etc/fstab. You can see an example on this 4GB KVM VPS, it will for non-Centmin Mod systems setup /tmp as tmpfs ram disk mount to 1/2 memory size = 2GB for /tmp. With Centmin Mod, it will detect <8GB memory system and setup a non-tmpfs non-ramdisk mount for /tmp that is larger size at 3.7GB so that apps tmp drive usage requirements don't put pressure on memory and are more adequate for /tmp usage.
    Code (Text):
    df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs  1.9G     0  1.9G   0% /dev
    tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
    tmpfs          tmpfs     2.0G  8.5M  1.9G   1% /run
    tmpfs          tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
    /dev/vda2      xfs        80G  7.1G   73G   9% /
    /dev/loop0     ext4      3.9G   76K  3.7G   1% /tmp
    tmpfs          tmpfs     390M     0  390M   0% /run/user/0
    


    But yes seems that is the expected outcome from what you outlined. The solution is to do a remount after a resize of the disk if you want to keep /etc/fstab tmp directory loop device entry. So resize and then remount. But are you saying normal server reboots (without disk resize) don't work with /etc/fstab /tmpdir loop device entry? Or only whenever you do a disk resize? If it's the latter, the correct way is to always do a remount after a disk resize to ensure /etc/fstab tmp directory loop device entry works and disk isn't in read only mode.

    Edit: better solution https://community.centminmod.com/th...ng-ro-file-system-on-reboot.23763/#post-96171
     
  3. Matt

    Matt Well-Known Member

    917
    413
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +664
    Local Time:
    11:25 PM
    1.5.15
    MariaDB 10.2
    Brand new, it was a new build VM this morning as well.

    XFS by default with DigitalOcean

    Code:
    # df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs  3.8G     0  3.8G   0% /dev
    tmpfs          tmpfs     3.8G     0  3.8G   0% /dev/shm
    tmpfs          tmpfs     3.8G   17M  3.8G   1% /run
    tmpfs          tmpfs     3.8G     0  3.8G   0% /sys/fs/cgroup
    /dev/vda1      xfs        80G   11G   70G  14% /
    tmpfs          tmpfs     777M     0  777M   0% /run/user/0
     
  4. Matt

    Matt Well-Known Member

    917
    413
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +664
    Local Time:
    11:25 PM
    1.5.15
    MariaDB 10.2
    Code:
    # cminfo versions
    
    1st:
    130.00beta01.b305 #Tue Apr 25 07:41:50 UTC 2023
    ..
    last 10:
    130.00beta01.b305 #Tue Apr 25 07:41:50 UTC 2023
    130.00beta01.b305 #Tue Apr 25 08:09:21 UTC 2023
    130.00beta01.b305 #Tue Apr 25 08:15:23 UTC 2023
    130.00beta01.b305 #Tue Apr 25 10:32:58 UTC 2023
     
  5. eva2000

    eva2000 Administrator Staff Member

    52,262
    12,005
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,508
    Local Time:
    8:25 AM
    Nginx 1.25.x
    MariaDB 10.x
    For read-only remount, try with
    Code (Text):
    mount -o remount,rw /
     
  6. Matt

    Matt Well-Known Member

    917
    413
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +664
    Local Time:
    11:25 PM
    1.5.15
    MariaDB 10.2
    Yep, that's what I've been having to do following the reboot, and then manually starting the services up.

    As soon as you reboot it again (for any purpose, even a straight reboot):

    Code:
    [11:39][root@tfc-website ~]# mount | grep "ro,"
    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
    /dev/vda1 on / type xfs (ro,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
     
  7. Matt

    Matt Well-Known Member

    917
    413
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +664
    Local Time:
    11:25 PM
    1.5.15
    MariaDB 10.2
    By the way, we aren't actually resizing the disk, we are only resizing the CPU / Memory, the disk remains untouched.
     
  8. eva2000

    eva2000 Administrator Staff Member

    52,262
    12,005
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,508
    Local Time:
    8:25 AM
    Nginx 1.25.x
    MariaDB 10.x
    try this in read-only state do the following
    Code (Text):
    mount -o remount,rw /
    mount -a
    

    and then try subsequent reboots and see if it's still read only
     
  9. Matt

    Matt Well-Known Member

    917
    413
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +664
    Local Time:
    11:25 PM
    1.5.15
    MariaDB 10.2
    No difference:

    Code:
    /usr/local/bin/dmotd: line 305: /root/centminlogs/cmm-login-git-checks_250423-115543.log: Read-only file system
    /usr/local/bin/dmotd: line 306: /root/centminlogs/cmm-login-git-checks_250423-115543.log: Read-only file system
    [11:55][root@tfc-website ~]# 
     
  10. eva2000

    eva2000 Administrator Staff Member

    52,262
    12,005
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,508
    Local Time:
    8:25 AM
    Nginx 1.25.x
    MariaDB 10.x
    If that doesn't work try this method instead

    create /usr/local/bin/mount-loop.sh shell script and give it executable permissions (chmod +x /usr/local/bin/mount-loop.sh)

    add to the file
    Code (Text):
    #!/bin/bash
    mount -o loop,rw,noexec,nosuid /home/usertmp_donotdelete /tmp
    


    Remove the /home/usertmp_donotdelete /tmp ext4 loop,rw,noexec,nosuid 0 0 entry from /etc/fstab.

    Create /etc/systemd/system/mount-loop.service with following content
    Code (Text):
    [Unit]
    Description=Mount loop device for /tmp
    
    [Service]
    ExecStart=/usr/local/bin/mount-loop.sh
    Type=oneshot
    RemainAfterExit=yes
    
    [Install]
    WantedBy=multi-user.target
    

    Then enable the mount-loop.service
    Code (Text):
    systemctl enable mount-loop.service


    Then reboot server and check df -hT command to see if /tmp is mounted and disk is not read-only
     
  11. Matt

    Matt Well-Known Member

    917
    413
    63
    May 25, 2014
    Rotherham, UK
    Ratings:
    +664
    Local Time:
    11:25 PM
    1.5.15
    MariaDB 10.2
    Code:
    [12:04][root@tfc-website ~]# df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs  1.8G     0  1.8G   0% /dev
    tmpfs          tmpfs     1.9G     0  1.9G   0% /dev/shm
    tmpfs          tmpfs     1.9G   17M  1.9G   1% /run
    tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
    /dev/vda1      xfs        80G   12G   69G  14% /
    /dev/loop0     ext4      3.9G  920K  3.7G   1% /tmp
    tmpfs          tmpfs     374M     0  374M   0% /run/user/0
    
     
  12. eva2000

    eva2000 Administrator Staff Member

    52,262
    12,005
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,508
    Local Time:
    8:25 AM
    Nginx 1.25.x
    MariaDB 10.x
    Looks like I need to update Centmin Mod's tmp directory mount routine to use systemd service :)
     
  13. eva2000

    eva2000 Administrator Staff Member

    52,262
    12,005
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,508
    Local Time:
    8:25 AM
    Nginx 1.25.x
    MariaDB 10.x
    @Matt can you try this updated /etc/systemd/system/mount-loop.service

    Code (Text):
    [Unit]
    Description=Mount loop device for /tmp
    After=local-fs.target
    Requires=local-fs.target
    
    [Service]
    ExecStart=/usr/local/bin/mount-loop.sh
    ExecStop=/usr/bin/umount /tmp
    Type=oneshot
    RemainAfterExit=yes
    
    [Install]
    WantedBy=multi-user.target
    


    then try
    Code (Text):
    systemctl daemon-reload
    systemctl enable mount-loop.service
    

    and then reboot your system and see if that works. This is probably what I will update for 130.00beta01 as it will also allow you to unmount the /tmp loop device too via
    Code (Text):
    systemctl stop mount-loop.service
    


    edit: works at least on fresh Upcloud 2CPU KVM VPS fresh install of private EL8 Centmin Mod 130.00beta01 with AlmaLinux 8.7
    Code (Text):
    systemctl status mount-loop
    ‚óŹ mount-loop.service - Mount loop device for /tmp
       Loaded: loaded (/etc/systemd/system/mount-loop.service; enabled; vendor preset: disabled)
       Active: active (exited) since Tue 2023-04-25 14:02:37 UTC; 7min ago
      Process: 700 ExecStart=/usr/local/bin/mount-loop.sh (code=exited, status=0/SUCCESS)
     Main PID: 700 (code=exited, status=0/SUCCESS)
        Tasks: 0 (limit: 24704)
       Memory: 0B
       CGroup: /system.slice/mount-loop.service
    Apr 25 14:02:37 almalinux-2cpu-4gb-us-nyc1 systemd[1]: Starting Mount loop device for /tmp...
    Apr 25 14:02:37 almalinux-2cpu-4gb-us-nyc1 systemd[1]: Started Mount loop device for /tmp.
    

    Code (Text):
    df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs  1.9G     0  1.9G   0% /dev
    tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
    tmpfs          tmpfs     2.0G  8.5M  1.9G   1% /run
    tmpfs          tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
    /dev/vda2      xfs        80G  7.1G   73G   9% /
    /dev/loop0     ext4      3.9G   76K  3.7G   1% /tmp
    tmpfs          tmpfs     390M     0  390M   0% /run/user/0
    
     
    Last edited: Apr 26, 2023