/ Register

Welcome to Centmin Mod Community
Register Now

AlmaLinux The server SWAP's at around 4:30am everyday

Discussion in 'Bug Reports' started by Simon Brown, Apr 15, 2025.

Tags:
Previous Thread Next Thread
Loading...
  1. Apr 15, 2025 #1
    Simon Brown

    Simon Brown Member

    57
    6
    8
    Feb 9, 2017
    Ratings:
    +18
    Local Time:
    11:07 AM
    1.11.9
    Okay, it's time to get to the bottom of this issue I've had on 10+ VPC's and 5 dedicated servers, all hosted between 5 different hosting companies. I've spent days trying to work out what's happening.

    The issue:
    Every day, the server(s) swap so I have to frequently run "swapoff -a", then "swapon -a" to clear the SWAP memory.

    It's been happening on Centos 7 and now on Alma 8. It starts from a fresh install. I've just spun up a new dedicated server, hardware raid 1 on 1TB NVMe's, 32 Gb memory with a few standard WordPress sites.

    As you can see, it's a new server with the latest (beta) Centmin installed, and standard installs as per the Centmin server setup guide. Yellow is cached memory.
    [​IMG]

    Swap is set to 0, which I know doesn't disable it, but it should only kick-in during extreme memory usage, right?:

    /etc/sysctl.d/101-sysctl.conf
    Code:
    vm.swappiness=0

    What seems to be happening is the cached memory isn't being cleared quickly enough when a scheduled task is run at around 4:30am every day (on this server), I have another VPC with a different host that this happens to at around 5am.

    32Gb of memory is overkill for the small sites running on the server at the moment.

    As an example, I cleared the memory cache using this command at around 7pm:
    Code:
    sync; echo 1 > /proc/sys/vm/drop_caches
    At around 4:30am a process starts that fills the memory with cache.
    [​IMG]

    I can't see anything in the crontab that starts at 4:30am:

    [​IMG]



    Is there another way to see what scheduled tasks are running, or even better, show what is running at 4:30am?

    Many thanks in advance for any help.

    • CentOS Version: AlmaLinux 8.10
    • Centmin Mod Version Installed: 140.00beta01
    • Nginx Version Installed: nginx version: 1.27.4
    • PHP Version Installed: PHP 7.4.33 (cli) (built: Mar 10 2025 13:40:55)
    • MariaDB MySQL Version Installed: 10.6.21-MariaDB
    • When was last time updated Centmin Mod code base ? :
      Centmin Mod local code is up to date at /usr/local/src/centminmod
      no available updates at this time...

    • Persistent Config:
      SELFSIGNEDSSL_ECDSA='y'
      PHPFINFO='y'
      PHP_OVERWRITECONF='n'
      PYTHON_INSTALL_ALTERNATIVES='y'
      NGXDYNAMIC_BROTLI='y'
      NGINX_LIBBROTLI='y'
      LETSENCRYPT_DETECT='y'
      DUALCERTS='y'
      SSL_PROTOCOL_MODERN='y'
      NGINX_VHOSTSTATS='n'
      MARCH_TARGETNATIVE='n'
      PHP_PGO='y'
      CF_DNSAPI_GLOBAL='y'
     
    Last edited: Apr 15, 2025
  2. Apr 16, 2025 at 9:46 PM #2
    eva2000

    eva2000 Administrator Staff Member

    55,165
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    8:07 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  3. Apr 16, 2025 at 11:45 PM #3
    Simon Brown

    Simon Brown Member

    57
    6
    8
    Feb 9, 2017
    Ratings:
    +18
    Local Time:
    11:07 AM
    1.11.9
    Thanks for these, the commands show the time the memory gets filled up, but I can't see what process is starting up and consuming all the memory.

    Is there a way to see the process running at 4/4:30am.

    Code:
    00:00:01    kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
03:30:01      3906556         0      0.00         0      0.00
03:35:01      3906556         0      0.00         0      0.00
03:40:01      3906556         0      0.00         0      0.00
03:45:01      3906556         0      0.00         0      0.00
03:50:01      3906556         0      0.00         0      0.00
03:55:01      3906556         0      0.00         0      0.00
04:00:01      3906556         0      0.00         0      0.00
04:05:01      3902716      3840      0.10      1576     41.04
04:10:01      3900156      6400      0.16      2756     43.06
04:15:01      3900156      6400      0.16      2764     43.19
04:20:01      3897084      9472      0.24      4312     45.52
04:25:01      3897084      9472      0.24      4480     47.30
04:30:01      3900156      6400      0.16      2972     46.44
04:35:01      3900156      6400      0.16      3040     47.50
04:40:01      3900156      6400      0.16      3040     47.50
04:45:01      3900156      6400      0.16      3060     47.81
04:50:01      3900156      6400      0.16      3072     48.00
04:55:01      3900156      6400      0.16      3072     48.00
05:00:01      3897084      9472      0.24      4768     50.34
05:05:01      3897084      9472      0.24      5080     53.63
05:10:01      3897084      9472      0.24      5124     54.10
    Many thanks for any help,
    Simon
     
  4. Apr 17, 2025 at 4:55 PM #4
    eva2000

    eva2000 Administrator Staff Member

    55,165
    12,249
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,825
    Local Time:
    8:07 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Try this shell script

    Code:
    # create 
nano ~/process_monitor.sh
# executable
chmod +x ~/process_monitor.sh
# backup cronjobs
crontab -l > backup.cronjobs.txt
# edit cronjobs set cronjob via command
# that invokes nano editor to edit you existing cronjobs
crontab -e
# setup cronjob as
* * * * * /home/your_username/process_monitor.sh
    Now, the script will run every minute. If the time is between 4:00 AM and 4:30 AM, it will append the pidstat output to a daily log file located in ~/logs/process_monitor/ so if running as root user would be at /root/logs/process_monitor/

    Code:
    #!/bin/bash

# === Configuration ===
# Set the directory where log files will be stored.
# Ensure this directory exists and the script has write permissions.
# Using $HOME/logs/ makes it user-specific and avoids permission issues with /var/log.
LOG_DIR="$HOME/logs/process_monitor"
# Set the log file name, including the date.
LOG_FILE="$LOG_DIR/process_usage_$(date +%Y-%m-%d).log"

# === Script Logic ===
# Create the log directory if it doesn't already exist.
# The -p flag ensures no error if the directory already exists and creates parent directories as needed.
mkdir -p "$LOG_DIR"

# Get the current time in HHMM format (e.g., 0415 for 4:15 AM).
CURRENT_TIME=$(date +%H%M)

# Define the start and end times for the monitoring window (4:00 AM to 4:30 AM).
# Times are in HHMM format.
START_TIME="0400"
END_TIME="0430" # Includes 4:30 AM exactly

# Check if the current time is within the desired window [START_TIME, END_TIME].
# Uses bash integer comparison operators (-ge for >=, -le for <=).
if [[ "$CURRENT_TIME" -ge "$START_TIME" && "$CURRENT_TIME" -le "$END_TIME" ]]; then
  # If within the window, log a timestamp indicating the start of this specific recording.
  echo "--- Recording processes at $(date '+%Y-%m-%d %H:%M:%S') ---" >> "$LOG_FILE"

  # Execute pidstat and append its output to the log file.
  # Options used:
  # -d: Report disk I/O statistics.
  # -u: Report CPU utilization.
  # -r: Report page faults and memory utilization.
  # -l: Display the full command line associated with the task.
  # -h: Display statistics in human-readable format (e.g., KB, MB).
  # 1 1: Take 1 sample over a 1-second interval to get a snapshot of current usage.
  pidstat -durlh 1 1 >> "$LOG_FILE"

  # Log an end timestamp for this recording interval.
  echo "--- End of recording interval ---" >> "$LOG_FILE"
  # Add a blank line for better readability between entries.
  echo "" >> "$LOG_FILE"
fi

# Exit successfully. Cron typically expects an exit code of 0 for success.
exit 0
     
Previous Thread Next Thread
Loading...

.