Sysadmin Mystery files in web root

what's output for command listing full file/directory permissions for /home/nginx/domains/demodomain.com/public ?

Code (Text):

ls -lah /home/nginx/domains/demodomain.com/public

output for these commands

Code (Text):

du -sh /home/nginx/domains
du -sh /var/lib/mysql

Code (Text):

df -hT

hmmm what other web sites and web app/scripts do you run on the server ? What version of those app/scripts are being used ? regularly updated ?

If you have wordpress or some other CMS, it could have been hacked and the infection spread to other parts of your server as Centmin Mod doesn't support site chroot/jailed environment. Though demodomain.com without a valid DNS setup routing demodomain.com to your server IP, it isn't web accessible directly.

But if they're in other site domain web roots, then those infections would be. Centmin Mod itself maybe secure but your web app/scripts may not be. Centmin Mod is provide as is, so short of scripted related bugs or issues, any further troubleshooting at the web app specific level are left to the Centmin Mod user to deal with. Unfortunately, cleaning up infected web app/scripts is something I would not support.

However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out

You'll need to post on the forums with the following info

• Server or VPS details ? XEN, KVM, OpenVZ, VMWare or dedicated server ? OS ? CentOS 6.9 or 7.4 ? 32bit or 64bit ?
• Web host ? any regular backups being made ?
• What version of Centmin Mod ? .07 stable or 08 stable or .09 beta01 or another branch version ?
• Was it fresh install or upgrade ?
• Method of install ? Via centmin.sh menu option 1, Git install or curl one liner install as outlined at centminmod.com/download.html ?
  
• How long ago did you install Centmin Mod ?
• When was last time you updated Centmin Mod via centmin.sh menu option 23 submenu option 2 ?
• Any other folks besides yourself have access to your server via root user, any sudo created users, any web app scripts have membership/user login to the apps i.e. forums, blogs etc ?

You can use cminfo versions command to list Centmin Mod version history from install time to present if you're using Centmin Mod 123.09beta01 and higher i.e.

Code (Text):

cminfo versions

Code (Text):

cminfo versions

1st:
1.2.3-eva2000.09.004 #Tue Apr  4 03:32:50 EDT 2017
..
last 10:
123.09beta01.b023 #Sat May 26 08:01:21 UTC 2018
123.09beta01.b023 #Sat May 26 08:14:15 UTC 2018
123.09beta01.b023 #Sat May 26 08:28:59 UTC 2018
123.09beta01.b023 #Sat May 26 08:32:26 UTC 2018
123.09beta01.b025 #Tue May 29 14:05:53 UTC 2018
123.09beta01.b025 #Tue May 29 14:09:59 UTC 2018
123.09beta01.b025 #Thu May 31 11:16:29 UTC 2018
123.09beta01.b032 #Wed Jul  4 01:17:15 UTC 2018
123.09beta01.b033 #Wed Jul  4 02:13:34 UTC 2018
123.09beta01.b033 #Wed Jul  4 02:45:25 UTC 2018

If on Centmin Mod 123.09beta01 or higher, you can also use cminfo netstat command to see what type of TCP connections are incoming/outgoing too see if there's any outbound connections that are common/frequent originating from hacked/inserted malware files

Code (Text):

cminfo netstat

Some suggestions

• centmin mod site and file structure overview at Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS may come in handy for backups/restores - knowing what to backup/restore or inspect
• centmin mod data migration guide Upgrade - Nginx - Redis - Insight Guide - Centmin Mod Site Data Migration Guide though in infected state, you'd be migrating the infected files too, so clean up first is a must
• you can check pure-ftpd logs, /var/log/secure, /var/log/message and /var/log/lfd.log for csf firewall login failure daemon for clues and if you have the hackers IP address should grep all logs to get an idea
  Code (Text):

  grep -R 'ipaddress' /var/log/*

• backup a copy of all those mysterious files you see, they can be useful in finding out what hackers possible did on your server or can potentially do. Alot of hackers via vulnerabilities can upload their own php shell with ftp like file management and viewing capabilities as well.
• With latest updated Centmin Mod code via centmin.sh menu option 23 submenu option 2 or 123.09beta01 via cmupdate command, install and run maldet malware scanner Maldet - Linux Malware Detect Addon (discussion) example run on demo site after maldet installed - maldet will quarantine any malware it thinks it has detected so you may find the files removed but maldet won't find 100% malware hence recommended to clean OS and reinstall. Running on demo site is a controlled scan as no other web app scripts are installed so whatever is reported is usually inserted by malware infections. After this scan, you can do a scan on /home/nginx/domains level for all sites directories
  
  Code (Text):

  maldet -u
  time maldet -a /home/nginx/domains/demodomain.com
  maldet --report list
  

  
  
  Code (Text):

  time maldet -a /home/nginx/domains/demodomain.com
  Linux Malware Detect v1.6.2
              (C) 2002-2017, R-fx Networks <proj@rfxn.com>
              (C) 2017, Ryan MacDonald <ryan@rfxn.com>
  This program may be freely redistributed under the terms of the GNU GPL v2
  
  maldet(10344): {scan} signatures loaded: 17085 (14276 MD5 | 2030 HEX | 779 YARA | 0 USER)
  maldet(10344): {scan} building file list for /home/nginx/domains/demodomain.com, this might take awhile...
  maldet(10344): {scan} setting nice scheduler priorities for all operations: cpunice 19 , ionice 6
  maldet(10344): {scan} file list completed in 0s, found 11 files...
  maldet(10344): {scan} found clamav binary at /usr/bin/clamscan, using clamav scanner engine...
  maldet(10344): {scan} scan of /home/nginx/domains/demodomain.com (11 files) in progress...
  
  maldet(10344): {scan} scan completed on /home/nginx/domains/demodomain.com: files 11, malware hits 0, cleaned hits 0, time 16s
  maldet(10344): {scan} scan report saved, to view run: maldet --report 180706-1912.10344
  
  real    0m16.018s
  user    0m14.998s
  sys     0m0.729s
  

  
  actual report invokes linux nano text editor so to exit from it CTRL+X
  
  Code (Text):

  maldet --report 180706-1912.10344
  
  HOST:      hostname
  SCAN ID:   180706-1912.10344
  STARTED:   Jul  6 2018 19:12:23 +0000
  COMPLETED: Jul  6 2018 19:12:39 +0000
  ELAPSED:   16s [find: 0s]
  
  PATH:          /home/nginx/domains/demodomain.com
  TOTAL FILES:   11
  TOTAL HITS:    0
  TOTAL CLEANED: 0
  
  ===============================================
  Linux Malware Detect v1.6.2 < proj@rfxn.com >
  

• check your site's logfiles outlined at Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS some log files may have logrotated and be compressed with gzip and end in .gz extension so command cat/tail commands won't work you can use zgrep or zcat to do the same thing for gzip files see example at Sysadmin - Disable log rotate?
• install addons/auditd.sh if you're on 123.09beta01 won't help with past infections but may help if infection is on-going and you keep getting reinfected - auditd you can track files and at least have some idea when and who/script wise is doing the infection provided you know auditd commands and understand the output - if you hire someone to look into this they will know how to use auditd so will help there
• maybe hire Sucuri folks for malware/infection removal and analysis Website Malware Removal - We Fix Hacked Websites | Sucuri if you use any of their listed CMS, wordpress, joomla, drupal, magento, phpbb and vbulletin and ask them if your web app/script isn't listed. If they're unfamiliar with Centmin Mod structure point them to Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS and this very post itself too.
• ultimately, it's highly recommended you restore from last known clean backup after a OS (centos) reinstall and start fresh. Usually I recommend to take a full backup of your hacked site/file state/server logs etc including file permissions and such so tar backup with preserved file/user permissions. So you can do further analysis later on to find out how you were infected etc

 

any files uploaded to both servers that are in common ? do you have phpmyadmin.sh addon installed ? or any manual adminer/phpmyadmin script installations too ?

FYI, install addons/auditd.sh on both servers as well

anything else in common ? i.e. using same pure-ftpd username/passwords ? same root user passwords ? same SSH keypair file/authentication ?

do both servers have access to each other via SSH or in anyway ? i.e. remote backups, remote mysql ? same web host ? same web host server ?

 

FYI after you install addons/auditd.sh, you'd need to setup a custom auditd rule as per Centmin Mod Auditd Support Added In Latest 123.09beta01 to monitor web root of /home/nginx/domains/demodomain.com/public to see and monitor how your files are getting re-infected (if you remove them /home/nginx/domains/demodomain.com/public/*) then monitor how they get re-added

something like creating rule file at /etc/audit/rules.d/demosite.rules and adding to it below auditd rule defined by key = demosite to monitor changes for -wa (write and access)

Code (Text):

-w /home/nginx/domains/demodomain.com/public -p wa -k demosite

then update auditd rules

Code (Text):

cd /usr/local/src/centminmod/addons/
./auditd.sh updaterules

Then you can search auditd logs by key = demosite when you see those infected files re-added

Code (Text):

ausearch -k demosite

 

@enderst might help Extending Maldet Malware Detect Linux & ClamAV Signatures

 

indeed seems to be related to wordpress/joomla/phpBB compromise of some kind from quick google search

 

Cheers @enderst was wondering what the progress was - Drupal - one of the commond attacked/compromised CMS like Wordpress ! Bad cooties