Discover Centmin Mod today
Register Now

Nginx Nginx 1.23.1 Upgrade issue

Discussion in 'Install & Upgrades or Pre-Install Questions' started by rdan, Sep 2, 2022.

  1. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
  2. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    looks like broken incomplete openssl 1.1.1q download
    Code (Text):
    
    tar (child): /svr-setup/openssl-1.1.1q.tar.gz: Cannot open: No such file or directory
    tar (child): Error is not recoverable: exiting now
    tar: Child returned status 2
    tar: Error is not recoverable: exiting now
    shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
    Assembler messages:
    Fatal error: can't create help-dummy.o: No such file or directory
    /usr/lib64/libatomic_ops.a found
    tar (child): /svr-setup/pcre-8.45.tar.gz: Cannot open: No such file or directory
    tar (child): Error is not recoverable: exiting now
    tar: Child returned status 2
    tar: Error is not recoverable: exiting now
    


    actually more downloads are broken
    Code (Text):
    tar (child): /svr-setup/srcache-nginx-module_v0.32.tar.gz: Cannot open: No such file or directory
    tar (child): Error is not recoverable: exiting now
    tar: Child returned status 2
    tar: Error is not recoverable: exiting now
    

    Code (Text):
    tar (child): /svr-setup/pcre-8.45.tar.gz: Cannot open: No such file or directory
    tar (child): Error is not recoverable: exiting now
    tar: Child returned status 2
    tar: Error is not recoverable: exiting now
    

    check your server's connectivity

    Also 124.00stable version is a few commits behind so make sure to run cmupdate first Commits · centminmod/centminmod
     
  3. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    But nginx update/recompile to 1.22.0 works fine.
     
  4. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    Is the DIR /svr-setup/ supposed to be empty always?
    Mine has nothing on it.
     
  5. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    nginx version wouldn't have affected this but I did notice your /svr-setup directory is ramdisk mounted as tmpfs. This is only temp setup at initial Centmin Mod install time if it detects if you have enough memory and at end of initial install should remove ramdisk /svr-setup version and copy files to disk based /svr-setup version via a unsetramdisk function. Looks like it didn't do this in your case.
    Code (Text):
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs   32G     0   32G   0% /dev
    tmpfs          tmpfs      32G     0   32G   0% /dev/shm
    tmpfs          tmpfs      32G  2.4G   29G   8% /run
    tmpfs          tmpfs      32G     0   32G   0% /sys/fs/cgroup
    /dev/md2       xfs       411G  320G   91G  78% /
    tmpfs          tmpfs      32G  100K   32G   1% /tmp
    /dev/nvme0n1p1 vfat      510M  3.6M  507M   1% /boot/efi
    tmpfs          tmpfs     6.3G     0  6.3G   0% /run/user/0
    tmpfs          tmpfs     2.2G     0  2.2G   0% /svr-setup
    

    noticed from nginx upgrade log complaining of /svr-setup device busy should that could be related
    Code (Text):
    tar (child): /svr-setup/srcache-nginx-module_v0.32.tar.gz: Cannot open: No such file or directory
    tar (child): Error is not recoverable: exiting now
    tar: Child returned status 2
    tar: Error is not recoverable: exiting now
    rm: cannot remove ‘/svr-setup/’: Device or resource busy
    


    You can manually fix this using these commands
    Code (Text):
    DIR_TMP=/svr-setup
    mkdir -p "${DIR_TMP}_disk"
    \cp -R ${DIR_TMP}/* "${DIR_TMP}_disk"
    umount -l "${DIR_TMP}"
    mkdir -p "${DIR_TMP}"
    \cp -R ${DIR_TMP}_disk/* "${DIR_TMP}"
    rm -rf "${DIR_TMP}_disk"
    

    verify with
    Code (Text):
    df -hT
     
    Last edited: Sep 2, 2022
  6. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    Ow, this server was installed Feb 2018, 4 years ago.
    Code:
    # df -hT
    Filesystem     Type      Size  Used Avail Use% Mounted on
    devtmpfs       devtmpfs   32G     0   32G   0% /dev
    tmpfs          tmpfs      32G     0   32G   0% /dev/shm
    tmpfs          tmpfs      32G  9.1M   32G   1% /run
    tmpfs          tmpfs      32G     0   32G   0% /sys/fs/cgroup
    /dev/md2       xfs       411G  320G   92G  78% /
    tmpfs          tmpfs      32G   44K   32G   1% /tmp
    /dev/nvme0n1p1 vfat      510M  3.6M  507M   1% /boot/efi
    tmpfs          tmpfs     6.3G     0  6.3G   0% /run/user/0
     
  7. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    interesting so the fix I suggested worked right?
     
  8. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    i don't have directory now.
     
  9. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    whoops missing a step so updated above post but for now just
    Code (Text):
    DIR_TMP=/svr-setup
    mkdir -p "${DIR_TMP}"
    
     
  10. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    Missing (")?
     
  11. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    Yeah typo fixed :D
     
  12. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    After running centmin, this directory is removed :(
    Still can't update nginx properly.

    All other server using same custom_config works fine though :}
     
  13. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    huh? what the?

    re-create the /svr-setup directory and then run centmin.sh debug mode via cmifno debug-menuexit command
    Code (Text):
    cminfo debug-menuexit
    

    should save a debug verbose log of a centmin.sh run and 24 exit at /root/centminlogs/debug-menuexit.txt
    Code (Text):
    cminfo debug-menuexit
    
    ------------------------------------------------------------------
    Debugging centmin.sh menu option 24 exit routine starting
    Please wait until complete...
    ------------------------------------------------------------------
    Full debug log saved at /root/centminlogs/debug-menuexit.txt
    
    Inspect Yum Check Times In debug-menuexit.txt log
    
    2022-09-01 19:49:48 + echo ' checking for YUM updates... please wait...'
    2022-09-01 19:49:48  checking for YUM updates... please wait...
    2022-09-01 19:49:48 +++ awk '/Updated: / {print $1,$2,$3}' /var/log/yum.log
    2022-09-01 19:49:48 +++ tail -1
    2022-09-01 19:49:48 ++ date -d 'Aug 31 19:47:30' +%s
    2022-09-01 19:49:48 + LASTYUMCHECK_EPOCHDATE=1661975250
    2022-09-01 19:49:48 ++ date +%s
    
    Yum Check Duration: 6.00 seconds
    
    ------------------------------------------------------------------
    Debugging centmin.sh menu option 24 exit routine completed
    ------------------------------------------------------------------
    

    sanitize and remove sensitive info from /root/centminlogs/debug-menuexit.txt and then share that log here or via PM
     
  14. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    As far as I can tell nothing running centmin.sh will delete /svr-setup from egrep filer
    Code (Text):
    cat /root/centminlogs/debug-menuexit.txt | egrep -C1 'svr-setup|DIR_TMP'
    2022-09-01 19:54:25 + HN=hostname
    2022-09-01 19:54:25 + DIR_TMP=/svr-setup
    2022-09-01 19:54:25 + CENTMINLOGDIR=/root/centminlogs
    --
    2022-09-01 19:54:25 + TMPFSLIMIT=4000000
    2022-09-01 19:54:25 + '[' '!' -d /svr-setup ']'
    2022-09-01 19:54:25 ++ cat /etc/resolv.conf
    --
    2022-09-01 19:54:25 + [[ -f /usr/local/bin/ccache ]]
    2022-09-01 19:54:25 + [[ ! -f /svr-setup/ccache-4.5.1.tar.gz ]]
    2022-09-01 19:54:25 + [[ -f /usr/local/bin/ccache ]]
    2022-09-01 19:54:25 + [[ -d /svr-setup/ccache-4.5.1 ]]
    2022-09-01 19:54:25 ++ ccache -V
    --
    2022-09-01 19:54:26 + [[ -f /usr/local/bin/axel ]]
    2022-09-01 19:54:26 + [[ ! -d /svr-setup/axel-2.16.1 ]]
    2022-09-01 19:54:26 + fileperm_fixes
    --
    2022-09-01 19:54:26 + ngxmodule_cleanups
    2022-09-01 19:54:26 + '[' -f /svr-setup/nginx-develkit_v0.2.19.tar.gz ']'
    2022-09-01 19:54:26 + '[' -d /svr-setup/ngx_devel_kit-0.2.19 ']'
    2022-09-01 19:54:26 + wpcompfix
    
     
  15. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    Let me refresh the code with this for now.
    cmupdate
    cmupdate update-stable
    cmdir
    centmin

    BRB.
     
  16. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    Sent unmodified code.
    Thanks Eva.
     
  17. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    There's nothing in the log that shows why /svr-setup is being deleted.

    You can install auditd via tools/auditd.sh https://community.centminmod.com/th...td-support-added-in-latest-123-09beta01.9071/

    Then setup an auditd rule for /svr-setup directory like https://community.centminmod.com/th...added-in-latest-123-09beta01.9071/#post-37761

    create and add file /etc/audit/rules.d/svr-setup.rules file with the rule below added setting auditd key = svrsetup
    Code (Text):
    -w /svr-setup -p wa -k svrsetup
    

    then run
    Code (Text):
    cd /usr/local/src/centminmod/tools
    ./auditd.sh updaterules
    

    then re-create /svr-setup
    Code (Text):
    mkdir -p /svr-setup
    

    then run centmin.sh menu and exit or your centmin.sh menu option 4 and exit

    then check auditd rule key svrsetup with command
    Code (Text):
    ausearch -k svrsetup
    

    and separate saved output for command
    Code (Text):
    ausearch -k svrsetup | aureport -f

    and then post output to a pastebin link as it may contain a lot of entries
     
    Last edited: Sep 2, 2022
  18. rdan

    rdan Premium Member Premium Member

    5,327
    1,337
    113
    May 25, 2014
    Ratings:
    +2,065
    Local Time:
    2:02 AM
    Mainline
    10.2
    This can be closed now (for this specific site and server) as I migrated to a new server.
    I can't debug it further.
     
  19. eva2000

    eva2000 Administrator Staff Member

    49,277
    11,294
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,571
    Local Time:
    4:02 AM
    Nginx 1.21.x
    MariaDB 10.x
    Good to hear it's resolved. Though was a strange issue!