Want to subscribe to topics you're interested in?
Become a Member

Sysadmin Mystery files in web root

Discussion in 'System Administration' started by enderst, Jul 7, 2018.

  1. enderst

    enderst New Member

    25
    5
    3
    Dec 12, 2017
    Ratings:
    +8
    Local Time:
    11:58 PM
    Finding files like these in site roots, not pasting contents. Anyone else seeing similar and able to stop it?
    Code:
    /home/nginx/domains/demodomain.com/public/article57.php
    /home/nginx/domains/demodomain.com/public/page36.php
    /home/nginx/domains/demodomain.com/public/alias.php
    /home/nginx/domains/demodomain.com/public/test74.php
    /home/nginx/domains/demodomain.com/public/xml.php
    /home/nginx/domains/demodomain.com/public/code.php
    /home/nginx/domains/demodomain.com/public/css.php
    /home/nginx/domains/demodomain.com/public/footer.php
    /home/nginx/domains/demodomain.com/public/files.php
    /home/nginx/domains/demodomain.com/public/dir66.php
    /home/nginx/domains/demodomain.com/public/inc.php
    /home/nginx/domains/demodomain.com/public/session79.php
    /home/nginx/domains/demodomain.com/public/global52.php
    /home/nginx/domains/demodomain.com/public/dirs53.php
    /home/nginx/domains/demodomain.com/public/functions34.php
    /home/nginx/domains/demodomain.com/public/plugin.php
    /home/nginx/domains/demodomain.com/public/page23.php
    /home/nginx/domains/demodomain.com/public/diff47.php
    /home/nginx/domains/demodomain.com/public/utf.php
    /home/nginx/domains/demodomain.com/public/plugin84.php
    /home/nginx/domains/demodomain.com/public/general.php
    /home/nginx/domains/demodomain.com/public/gallery74.php
    /home/nginx/domains/demodomain.com/public/cache52.php
    

     
  2. eva2000

    eva2000 Administrator Staff Member

    40,322
    8,930
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,753
    Local Time:
    3:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    what's output for command listing full file/directory permissions for /home/nginx/domains/demodomain.com/public ?
    Code (Text):
    ls -lah /home/nginx/domains/demodomain.com/public
    

    output for these commands
    Code (Text):
    du -sh /home/nginx/domains
    du -sh /var/lib/mysql
    

    Code (Text):
    df -hT
    


    hmmm what other web sites and web app/scripts do you run on the server ? What version of those app/scripts are being used ? regularly updated ?

    If you have wordpress or some other CMS, it could have been hacked and the infection spread to other parts of your server as Centmin Mod doesn't support site chroot/jailed environment. Though demodomain.com without a valid DNS setup routing demodomain.com to your server IP, it isn't web accessible directly.

    But if they're in other site domain web roots, then those infections would be. Centmin Mod itself maybe secure but your web app/scripts may not be. Centmin Mod is provide as is, so short of scripted related bugs or issues, any further troubleshooting at the web app specific level are left to the Centmin Mod user to deal with. Unfortunately, cleaning up infected web app/scripts is something I would not support.

    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)

    You'll need to post on the forums with the following info
    • Server or VPS details ? XEN, KVM, OpenVZ, VMWare or dedicated server ? OS ? CentOS 6.9 or 7.4 ? 32bit or 64bit ?
    • Web host ? any regular backups being made ?
    • What version of Centmin Mod ? .07 stable or 08 stable or .09 beta01 or another branch version ?
    • Was it fresh install or upgrade ?
    • Method of install ? Via centmin.sh menu option 1, Git install or curl one liner install as outlined at centminmod.com/download.html ?
    • How long ago did you install Centmin Mod ?
    • When was last time you updated Centmin Mod via centmin.sh menu option 23 submenu option 2 ?
    • Any other folks besides yourself have access to your server via root user, any sudo created users, any web app scripts have membership/user login to the apps i.e. forums, blogs etc ?
    You can use cminfo versions command to list Centmin Mod version history from install time to present if you're using Centmin Mod 123.09beta01 and higher i.e.
    Code (Text):
    cminfo versions
    

    Code (Text):
    cminfo versions
    
    1st:
    1.2.3-eva2000.09.004 #Tue Apr  4 03:32:50 EDT 2017
    ..
    last 10:
    123.09beta01.b023 #Sat May 26 08:01:21 UTC 2018
    123.09beta01.b023 #Sat May 26 08:14:15 UTC 2018
    123.09beta01.b023 #Sat May 26 08:28:59 UTC 2018
    123.09beta01.b023 #Sat May 26 08:32:26 UTC 2018
    123.09beta01.b025 #Tue May 29 14:05:53 UTC 2018
    123.09beta01.b025 #Tue May 29 14:09:59 UTC 2018
    123.09beta01.b025 #Thu May 31 11:16:29 UTC 2018
    123.09beta01.b032 #Wed Jul  4 01:17:15 UTC 2018
    123.09beta01.b033 #Wed Jul  4 02:13:34 UTC 2018
    123.09beta01.b033 #Wed Jul  4 02:45:25 UTC 2018
    


    If on Centmin Mod 123.09beta01 or higher, you can also use cminfo netstat command to see what type of TCP connections are incoming/outgoing too see if there's any outbound connections that are common/frequent originating from hacked/inserted malware files
    Code (Text):
    cminfo netstat
    


    Some suggestions
    • centmin mod site and file structure overview at Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS may come in handy for backups/restores - knowing what to backup/restore or inspect
    • centmin mod data migration guide Upgrade - Nginx - Redis - Insight Guide - Centmin Mod Site Data Migration Guide though in infected state, you'd be migrating the infected files too, so clean up first is a must
    • you can check pure-ftpd logs, /var/log/secure, /var/log/message and /var/log/lfd.log for csf firewall login failure daemon for clues and if you have the hackers IP address should grep all logs to get an idea
      Code (Text):
      grep -R 'ipaddress' /var/log/*
    • backup a copy of all those mysterious files you see, they can be useful in finding out what hackers possible did on your server or can potentially do. Alot of hackers via vulnerabilities can upload their own php shell with ftp like file management and viewing capabilities as well.
    • With latest updated Centmin Mod code via centmin.sh menu option 23 submenu option 2 or 123.09beta01 via cmupdate command, install and run maldet malware scanner Maldet - Linux Malware Detect Addon (discussion) example run on demo site after maldet installed - maldet will quarantine any malware it thinks it has detected so you may find the files removed but maldet won't find 100% malware hence recommended to clean OS and reinstall. Running on demo site is a controlled scan as no other web app scripts are installed so whatever is reported is usually inserted by malware infections. After this scan, you can do a scan on /home/nginx/domains level for all sites directories
      Code (Text):
      maldet -u
      time maldet -a /home/nginx/domains/demodomain.com
      maldet --report list
      

      Code (Text):
      time maldet -a /home/nginx/domains/demodomain.com
      Linux Malware Detect v1.6.2
                  (C) 2002-2017, R-fx Networks <[email protected]>
                  (C) 2017, Ryan MacDonald <[email protected]>
      This program may be freely redistributed under the terms of the GNU GPL v2
      
      maldet(10344): {scan} signatures loaded: 17085 (14276 MD5 | 2030 HEX | 779 YARA | 0 USER)
      maldet(10344): {scan} building file list for /home/nginx/domains/demodomain.com, this might take awhile...
      maldet(10344): {scan} setting nice scheduler priorities for all operations: cpunice 19 , ionice 6
      maldet(10344): {scan} file list completed in 0s, found 11 files...
      maldet(10344): {scan} found clamav binary at /usr/bin/clamscan, using clamav scanner engine...
      maldet(10344): {scan} scan of /home/nginx/domains/demodomain.com (11 files) in progress...
      
      maldet(10344): {scan} scan completed on /home/nginx/domains/demodomain.com: files 11, malware hits 0, cleaned hits 0, time 16s
      maldet(10344): {scan} scan report saved, to view run: maldet --report 180706-1912.10344
      
      real    0m16.018s
      user    0m14.998s
      sys     0m0.729s
      

      actual report invokes linux nano text editor so to exit from it CTRL+X
      Code (Text):
      maldet --report 180706-1912.10344
      
      HOST:      hostname
      SCAN ID:   180706-1912.10344
      STARTED:   Jul  6 2018 19:12:23 +0000
      COMPLETED: Jul  6 2018 19:12:39 +0000
      ELAPSED:   16s [find: 0s]
      
      PATH:          /home/nginx/domains/demodomain.com
      TOTAL FILES:   11
      TOTAL HITS:    0
      TOTAL CLEANED: 0
      
      ===============================================
      Linux Malware Detect v1.6.2 < [email protected] >
      
    • check your site's logfiles outlined at Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS some log files may have logrotated and be compressed with gzip and end in .gz extension so command cat/tail commands won't work you can use zgrep or zcat to do the same thing for gzip files see example at Sysadmin - Disable log rotate?
    • install addons/auditd.sh if you're on 123.09beta01 won't help with past infections but may help if infection is on-going and you keep getting reinfected - auditd you can track files and at least have some idea when and who/script wise is doing the infection provided you know auditd commands and understand the output - if you hire someone to look into this they will know how to use auditd so will help there
    • maybe hire Sucuri folks for malware/infection removal and analysis Website Malware Removal - We Fix Hacked Websites | Sucuri if you use any of their listed CMS, wordpress, joomla, drupal, magento, phpbb and vbulletin and ask them if your web app/script isn't listed. If they're unfamiliar with Centmin Mod structure point them to Centmin Mod Configuration Files - CentminMod.com LEMP Nginx web stack for CentOS and this very post itself too.
    • ultimately, it's highly recommended you restore from last known clean backup after a OS (centos) reinstall and start fresh. Usually I recommend to take a full backup of your hacked site/file state/server logs etc including file permissions and such so tar backup with preserved file/user permissions. So you can do further analysis later on to find out how you were infected etc
     
  3. enderst

    enderst New Member

    25
    5
    3
    Dec 12, 2017
    Ratings:
    +8
    Local Time:
    11:58 PM
    Thanks for the input. I wasn't requesting cleanup or pointing fingers, was just looking to see if anyone else had this happen to them and resolved it. This server is all Wordpress running on the latest centminmod beta code. I have Maldet running with inotify and it didn't see anything interesting and nothing with a manual -a scan. Once i sort it out I'll post details.
    Concerning that demodomain got hit. I have another server running centminmod beta code with a php site not a CMS. Found some of those nifty files in it as well, only a couple but still a couple too many.
    Thanks for the auditd tip, I'll get that setup.
     
    • Informative Informative x 1
  4. eva2000

    eva2000 Administrator Staff Member

    40,322
    8,930
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,753
    Local Time:
    3:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    any files uploaded to both servers that are in common ? do you have phpmyadmin.sh addon installed ? or any manual adminer/phpmyadmin script installations too ?

    FYI, install addons/auditd.sh on both servers as well

    anything else in common ? i.e. using same pure-ftpd username/passwords ? same root user passwords ? same SSH keypair file/authentication ?

    do both servers have access to each other via SSH or in anyway ? i.e. remote backups, remote mysql ? same web host ? same web host server ?
     
  5. eva2000

    eva2000 Administrator Staff Member

    40,322
    8,930
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,753
    Local Time:
    3:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    FYI after you install addons/auditd.sh, you'd need to setup a custom auditd rule as per Centmin Mod Auditd Support Added In Latest 123.09beta01 to monitor web root of /home/nginx/domains/demodomain.com/public to see and monitor how your files are getting re-infected (if you remove them /home/nginx/domains/demodomain.com/public/*) then monitor how they get re-added

    something like creating rule file at /etc/audit/rules.d/demosite.rules and adding to it below auditd rule defined by key = demosite to monitor changes for -wa (write and access)
    Code (Text):
    -w /home/nginx/domains/demodomain.com/public -p wa -k demosite
    

    then update auditd rules
    Code (Text):
    cd /usr/local/src/centminmod/addons/
    ./auditd.sh updaterules
    

    Then you can search auditd logs by key = demosite when you see those infected files re-added
    Code (Text):
    ausearch -k demosite
    
     
  6. eva2000

    eva2000 Administrator Staff Member

    40,322
    8,930
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,753
    Local Time:
    3:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  7. Meirami

    Meirami Member

    128
    15
    18
    Dec 21, 2017
    Ratings:
    +41
    Local Time:
    8:58 AM
    Google gave few pages with similar files, when I search session79.php.
     
  8. eva2000

    eva2000 Administrator Staff Member

    40,322
    8,930
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,753
    Local Time:
    3:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    indeed seems to be related to wordpress/joomla/phpBB compromise of some kind from quick google search
     
  9. enderst

    enderst New Member

    25
    5
    3
    Dec 12, 2017
    Ratings:
    +8
    Local Time:
    11:58 PM
    Sorry I feel like I left you hanging.
    Here is an example of what I was seeing but I think I was hit with two different exploits.
    Drupal ico file hack, index.php hack

    My problem from the start was these were sites migrated and they migrated with their cooties.

    Good news is they're clean now.

    Thanks for all of the info and suggestions.
     
    • Informative Informative x 1
  10. eva2000

    eva2000 Administrator Staff Member

    40,322
    8,930
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,753
    Local Time:
    3:58 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Cheers @enderst was wondering what the progress was - Drupal - one of the commond attacked/compromised CMS like Wordpress ! Bad cooties :yuck:
     
..