Security was my server compromised?

Please fill in any relevant information that applies to you:

• CentOS Version: i.e. CentOS 6 32bit or 64bit / CentOS 7 64bit ?
• Centmin Mod Version Installed: i.e. 123.08stable or 123.09beta01
• Nginx Version Installed: i.e. 1.15.3
• PHP Version Installed: i.e. 5.6.37, 7.0.31, 7.1.21, 7.2.9
• MariaDB MySQL Version Installed: i.e. 10.0.x or 10.1.xx or 10.2.xx
• When was last time updated Centmin Mod code base ? : i.e. run centmin.sh menu option 23 submenu option 2 or cmupdate command
• Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
  
  Code (Text):

  cat /etc/centminmod/custom_config.inc
  

  
  Post output in CODE tags.

what is the output for these commands

Code (Text):

df -hT
virt-what
ls -lah /var/spool/cron
ls -lah /var/spool

and

Code (Text):

csf -ra

and

Code (Text):

service lfd status| sed -e "s|$(hostname)|hostname|g"

and

Code (Text):

service csf status| sed -e "s|$(hostname)|hostname|g"

and

Code (Text):

perl /etc/csf/csftest.pl

for posting code or output from commands to keep the formatting, you might want to use CODE tags for code How to use forum BBCODE code tags

 

how are you creating cronjob ?
use crontab -l to view current cronjobs and crontab -e command to invoke nano text editor see guide at HowTo: Add Jobs To cron Under Linux or UNIX?

you can't restart csf as you have it disabled, to reneable run

Code (Text):

csf -e

 

you need to add cronjobs yourself via crontab -e command as it launches nano linux text editor where you add your cronjobs yourself - that is proper way of adding cronjobs

how are you attempting to add cronjobs ? not via adding them to crontab -e launched nano linux editor ?

post output from command

Code (Text):

csf -e

 

i noticed from above output listing for /var/spool/cron, that you're missing root user's cron file at /var/spool/cron/root. Did you try editing /var/spool/cron/root directly ? that's a big no no, crontab -e is correct way to edit cronjobs for root user. So what you have done is probably accidentally delete /var/spool/cron/root file when you tried to incorrectly edit /var/spool/cron/root directly

so re-create the file /var/spool/cron/root via commands below

Code (Text):

touch /var/spool/cron/root
chmod 600 /var/spool/cron/root

then using crontab -e command launch nano linux editor and re-add the default centmin mod cronjobs below and save and exit nano via ctrl+x and answer yes to save to exit - upon exiting crontab -e, your new cronjobs will register and be viewable via crontab -l command

Code (Text):

13 23 * * * /usr/local/src/centminmod/tools/autoprotect.sh >/dev/null 2>&1
0 */4 * * * /usr/bin/cminfo_updater 2>/dev/null

if you had other cronjobs setup properly, but are not lost when /var/spool/cron/root was deleted, then you can use cronjob log at /var/log/cron to get an idea what those cronjob commands were so to re-create them via crontab -e

i.e. list last 100 lines of cronjob log at /var/log/cron

Code (Text):

tail -100 /var/log/cron

if you had centmin.sh menu option 22 wordpress sites setup, there's a backup of their created cronjobs in files listed at /home/nginx/domains/${vhostname}/cronjobs/ where ${vhostname} is your wordpress domain name. Use crontab -e to re-add those backed up wordpress cronjobs too.

 

have you done this step first

 

those steps to fix cronjob not csf so separate issues or could be related what is output from these commands
inspecting line 3702 of /usr/sbin/csf which is called from running or creating /etc/cron.d/csf_update and filter grep on keyword 3702 with printing context of 5 lines before and after keyword match

Code (Text):

cat -n /usr/sbin/csf | grep -C5 3702

also output to check if /etc/cron.d directory exists, you could have accidentally deleted that directory ?

Code (Text):

ls -lah /etc | grep cron
ls -lah /etc/cron.d

example of expected output on working system

Code (Text):

ls -lah /etc | grep cron
-rw-------    1 root  root           541 Nov 20  2018 anacrontab
drwxr-xr-x.   2 root  root          4.0K Sep  5 12:40 cron.d
drwxr-xr-x.   2 root  root          4.0K Jul 16 07:00 cron.daily
-rw-------    1 root  root             0 Nov 20  2018 cron.deny
drwxr-xr-x.   2 root  root          4.0K Jan 18  2019 cron.hourly
drwxr-xr-x.   2 root  root          4.0K Jun  9  2014 cron.monthly
-rw-r--r--    1 root  root           466 Jan 18  2019 crontab
drwxr-xr-x.   2 root  root          4.0K Jun  9  2014 cron.weekly

Code (Text):

ls -lah /etc/cron.d
total 44K
drwxr-xr-x.   2 root root 4.0K Sep  5 12:40 .
drwxr-xr-x. 121 root root  12K Sep  5 12:40 ..
-rw-r--r--    1 root root  128 Nov 20  2018 0hourly
-rw-r--r--    1 root root   14 Feb  1  2013 csf-cron
-rw-------    1 root root   46 Jan 18  2019 csf_update
-rw-r--r--    1 root root   74 Aug 24 07:00 lfd-cron
-rw-r--r--    1 root root  108 Oct 30  2018 raid-check
-rw-------    1 root root  233 Sep  5 12:40 sysstat

to double check if any commands you ran accidentally deleted files or directories you can inspect command history log filter egrep for words cron and rm and then pipe results into another egrep filter to return only august and september date timestamped entries for past 2 months

Code (Text):

history | egrep 'cron|rm ' | egrep '.09.19|.08.19'

 

nginx.exe is windows executable so doesn't belong on centos linux and if those are empty them someone deleted them and if you didn't install docker that that is worrying

what about history output ?

 

hmm could be sign of being hacked/compromised as the hacker would wipe history logs to hide traces of their commands etc but also can be case if you accidentally ran history -c command to clear history as well

How long did you have csf firewall disabled for ? by disabling csf firewall you could of allowed a hacker to penetrate any open unprotected additional services you installed after installing centmin mod too. But hard to say what happened from the outside of your server from our perspective

what is output for

Code (Text):

docker ps -a

Code (Text):

docker images

Code (Text):

docker info

Code (Text):

ps aufxw

and to check when docker containers were started/created

Code (Text):

docker ps -a -q | while read id; do docker inspect --format='{{.Created}} {{.Config.Image}} {{.State.StartedAt}}' $id; done