Sysadmin The server SWAP's at around 4:30am everyday

Simon Brown · Apr 15, 2025

Okay, it's time to get to the bottom of this issue I've had on 10+ VPC's and 5 dedicated servers, all hosted between 5 different hosting companies. I've spent days trying to work out what's happening.

The issue:
Every day, the server(s) swap so I have to frequently run "swapoff -a", then "swapon -a" to clear the SWAP memory.

It's been happening on Centos 7 and now on Alma 8. It starts from a fresh install. I've just spun up a new dedicated server, hardware raid 1 on 1TB NVMe's, 32 Gb memory with a few standard WordPress sites.

As you can see, it's a new server with the latest (beta) Centmin installed, and standard installs as per the Centmin server setup guide. Yellow is cached memory.

Swap is set to 0, which I know doesn't disable it, but it should only kick-in during extreme memory usage, right?:

/etc/sysctl.d/101-sysctl.conf
Code:
vm.swappiness=0
What seems to be happening is the cached memory isn't being cleared quickly enough when a scheduled task is run at around 4:30am every day (on this server), I have another VPC with a different host that this happens to at around 5am.

32Gb of memory is overkill for the small sites running on the server at the moment.

As an example, I cleared the memory cache using this command at around 7pm:
Code:
sync; echo 1 > /proc/sys/vm/drop_caches
At around 4:30am a process starts that fills the memory with cache.

I can't see anything in the crontab that starts at 4:30am:

Is there another way to see what scheduled tasks are running, or even better, show what is running at 4:30am?

Many thanks in advance for any help.

CentOS Version: AlmaLinux 8.10

Centmin Mod Version Installed: 140.00beta01

Nginx Version Installed: nginx version: 1.27.4

PHP Version Installed: PHP 7.4.33 (cli) (built: Mar 10 2025 13:40:55)

MariaDB MySQL Version Installed: 10.6.21-MariaDB

When was last time updated Centmin Mod code base ? :
Centmin Mod local code is up to date at /usr/local/src/centminmod
no available updates at this time...

Persistent Config:
SELFSIGNEDSSL_ECDSA='y'
PHPFINFO='y'
PHP_OVERWRITECONF='n'
PYTHON_INSTALL_ALTERNATIVES='y'
NGXDYNAMIC_BROTLI='y'
NGINX_LIBBROTLI='y'
LETSENCRYPT_DETECT='y'
DUALCERTS='y'
SSL_PROTOCOL_MODERN='y'
NGINX_VHOSTSTATS='n'
MARCH_TARGETNATIVE='n'
PHP_PGO='y'
CF_DNSAPI_GLOBAL='y'

eva2000 · Apr 16, 2025

cminfo command has some flags that worth looking at https://community.centminmod.com/threads/cminfo-command-explained.11399/

https://community.centminmod.com/threads/cminfo-command-explained.11399/#post-80764
Code (Text):
cminfo top
https://community.centminmod.com/threads/cminfo-command-explained.11399/#post-80817
Code (Text):
cminfo sar-cpu
cminfo sar-mem
and sysstat sar command directly

for memory
Code (Text):
sar -r
for swap
Code (Text):
sar -S

Simon Brown · Apr 16, 2025

Thanks for these, the commands show the time the memory gets filled up, but I can't see what process is starting up and consuming all the memory.

Is there a way to see the process running at 4/4:30am.

Code:

00:00:01    kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
03:30:01      3906556         0      0.00         0      0.00
03:35:01      3906556         0      0.00         0      0.00
03:40:01      3906556         0      0.00         0      0.00
03:45:01      3906556         0      0.00         0      0.00
03:50:01      3906556         0      0.00         0      0.00
03:55:01      3906556         0      0.00         0      0.00
04:00:01      3906556         0      0.00         0      0.00
04:05:01      3902716      3840      0.10      1576     41.04
04:10:01      3900156      6400      0.16      2756     43.06
04:15:01      3900156      6400      0.16      2764     43.19
04:20:01      3897084      9472      0.24      4312     45.52
04:25:01      3897084      9472      0.24      4480     47.30
04:30:01      3900156      6400      0.16      2972     46.44
04:35:01      3900156      6400      0.16      3040     47.50
04:40:01      3900156      6400      0.16      3040     47.50
04:45:01      3900156      6400      0.16      3060     47.81
04:50:01      3900156      6400      0.16      3072     48.00
04:55:01      3900156      6400      0.16      3072     48.00
05:00:01      3897084      9472      0.24      4768     50.34
05:05:01      3897084      9472      0.24      5080     53.63
05:10:01      3897084      9472      0.24      5124     54.10

Many thanks for any help,
Simon

eva2000 · Apr 17, 2025

Try this shell script

Code:

# create 
nano ~/process_monitor.sh
# executable
chmod +x ~/process_monitor.sh
# backup cronjobs
crontab -l > backup.cronjobs.txt
# edit cronjobs set cronjob via command
# that invokes nano editor to edit you existing cronjobs
crontab -e
# setup cronjob as
* * * * * /home/your_username/process_monitor.sh

Now, the script will run every minute. If the time is between 4:00 AM and 4:30 AM, it will append the pidstat output to a daily log file located in ~/logs/process_monitor/ so if running as root user would be at /root/logs/process_monitor/

Code:

#!/bin/bash

# === Configuration ===
# Set the directory where log files will be stored.
# Ensure this directory exists and the script has write permissions.
# Using $HOME/logs/ makes it user-specific and avoids permission issues with /var/log.
LOG_DIR="$HOME/logs/process_monitor"
# Set the log file name, including the date.
LOG_FILE="$LOG_DIR/process_usage_$(date +%Y-%m-%d).log"

# === Script Logic ===
# Create the log directory if it doesn't already exist.
# The -p flag ensures no error if the directory already exists and creates parent directories as needed.
mkdir -p "$LOG_DIR"

# Get the current time in HHMM format (e.g., 0415 for 4:15 AM).
CURRENT_TIME=$(date +%H%M)

# Define the start and end times for the monitoring window (4:00 AM to 4:30 AM).
# Times are in HHMM format.
START_TIME="0400"
END_TIME="0430" # Includes 4:30 AM exactly

# Check if the current time is within the desired window [START_TIME, END_TIME].
# Uses bash integer comparison operators (-ge for >=, -le for <=).
if [[ "$CURRENT_TIME" -ge "$START_TIME" && "$CURRENT_TIME" -le "$END_TIME" ]]; then
  # If within the window, log a timestamp indicating the start of this specific recording.
  echo "--- Recording processes at $(date '+%Y-%m-%d %H:%M:%S') ---" >> "$LOG_FILE"

  # Execute pidstat and append its output to the log file.
  # Options used:
  # -d: Report disk I/O statistics.
  # -u: Report CPU utilization.
  # -r: Report page faults and memory utilization.
  # -l: Display the full command line associated with the task.
  # -h: Display statistics in human-readable format (e.g., KB, MB).
  # 1 1: Take 1 sample over a 1-second interval to get a snapshot of current usage.
  pidstat -durlh 1 1 >> "$LOG_FILE"

  # Log an end timestamp for this recording interval.
  echo "--- End of recording interval ---" >> "$LOG_FILE"
  # Add a blank line for better readability between entries.
  echo "" >> "$LOG_FILE"
fi

# Exit successfully. Cron typically expects an exit code of 0 for success.
exit 0

Simon Brown · Apr 23, 2025

Thank you so much for taking the time to create and share this code. I've set it all up. I'll report back asap.

Log in / Register

Forums

Welcome to Centmin Mod Community

Sysadmin The server SWAP's at around 4:30am everyday

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

.

Log in / Register

Useful Searches

Welcome to Centmin Mod Community

Sysadmin The server SWAP's at around 4:30am everyday

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

eva2000 Administrator Staff Member

Simon Brown Premium Member Premium Member

.