Discover Centmin Mod today
Register Now

Security was my server compromised?

Discussion in 'System Administration' started by yunos, Sep 15, 2019.

  1. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    csf cannot be started and i noticed
    /var/spool/cron was changed yesterday at 8 pm and as root i cant even chmode or create any file in the directory. is my server infected with malware or its something else?
     
  2. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Please fill in any relevant information that applies to you:
    • CentOS Version: i.e. CentOS 6 32bit or 64bit / CentOS 7 64bit ?
    • Centmin Mod Version Installed: i.e. 123.08stable or 123.09beta01
    • Nginx Version Installed: i.e. 1.15.3
    • PHP Version Installed: i.e. 5.6.37, 7.0.31, 7.1.21, 7.2.9
    • MariaDB MySQL Version Installed: i.e. 10.0.x or 10.1.xx or 10.2.xx
    • When was last time updated Centmin Mod code base ? : i.e. run centmin.sh menu option 23 submenu option 2 or cmupdate command
    • Persistent Config: Do you have any persistent config file options set in /etc/centminmod/custom_config.inc ? You can check via this command:
      Code (Text):
      cat /etc/centminmod/custom_config.inc
      

      Post output in CODE tags.

    what is the output for these commands
    Code (Text):
    df -hT
    virt-what
    ls -lah /var/spool/cron
    ls -lah /var/spool
    

    and
    Code (Text):
    csf -ra
    

    and
    Code (Text):
    service lfd status| sed -e "s|$(hostname)|hostname|g"
    

    and
    Code (Text):
    service csf status| sed -e "s|$(hostname)|hostname|g"
    

    and
    Code (Text):
    perl /etc/csf/csftest.pl
    

    for posting code or output from commands to keep the formatting, you might want to use CODE tags for code How to use forum BBCODE code tags :)
     
    • Like Like x 1
  3. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    centos 7 64bit
    123.09beta01
    nginx 1.17.3
    php 7.3.9
    mariadb 10.4.7
    cmupdate was executed 2 weeks ago.

    note that i also cant start csf due to getting perrmision denied to create in cron
    Error: Could not create /etc/cron.d/csf_update: Permission denied, at line 3702


    i have persistent config for centmin
    Code:
    LETSENCRYPT_DETECT='y'
    LIBRESSL_SWITCH='n'
    KEYLENGTH='ec-256' 
    Code:
    Filesystem          Type      Size  Used Avail Use% Mounted on
    devtmpfs            devtmpfs   12G     0   12G   0% /dev
    /dev/mapper/vg-root ext4      207G   13G  184G   7% / 
    virt-what
    resuilts in no respons

    Code:
    ls -lah /var/spool/cron
    total 8.0K
    drwx------. 2 root root 4.0K Sep 14 18:33 .
    drwxr-xr-x. 9 root root 4.0K Sep  5 14:10 .. 
    Code:
    ls -lah /var/spool
    total 36K
    drwxr-xr-x.  9 root  root 4.0K Sep  5 14:10 .
    drwxr-xr-x. 21 root  root 4.0K Sep 14 18:33 ..
    drwxr-xr-x.  2 root  root 4.0K Nov 20  2018 anacron
    drwx------.  2 root  root 4.0K Sep 14 18:33 cron
    drwx------   2 cyrus mail 4.0K Aug  4  2017 imap
    drwxr-xr-x.  2 root  root 4.0K Apr 11  2018 lpd
    drwxrwxr-x.  2 root  mail 4.0K Sep 14 19:26 mail
    drwxr-xr-x.  2 root  root 4.0K Apr 13  2018 plymouth
    drwxr-xr-x. 16 root  root 4.0K Sep  5 14:17 postfix 
    Code:
    service lfd status| sed -e "s|$(hostname)|hostname|g"
    Redirecting to /bin/systemctl status lfd.service
    ● lfd.service - ConfigServer Firewall & Security - lfd
       Loaded: loaded (/usr/lib/systemd/system/lfd.service; enabled; vendor preset: disabled)
       Active: failed (Result: signal) since Sun 2019-09-15 05:28:20 UTC; 8s ago
      Process: 8217 ExecStart=/usr/sbin/lfd (code=exited, status=0/SUCCESS)
     Main PID: 8234 (code=killed, signal=KILL)
    
    Sep 15 05:28:20 hostname systemd[1]: Starting ConfigServer Firewall & Security - lfd...
    Sep 15 05:28:20 hostname systemd[1]: Started ConfigServer Firewall & Security - lfd.
    Sep 15 05:28:20 hostname systemd[1]: lfd.service: main process exited, code=killed, status=9/KILL
    Sep 15 05:28:20 hostname systemd[1]: Unit lfd.service entered failed state.
    Sep 15 05:28:20 hostname systemd[1]: lfd.service failed. 
    Code:
    service csf status| sed -e "s|$(hostname)|hostname|g"
    Redirecting to /bin/systemctl status csf.service
    ● csf.service - ConfigServer Firewall & Security - csf
       Loaded: loaded (/usr/lib/systemd/system/csf.service; enabled; vendor preset: disabled)
       Active: failed (Result: exit-code) since Fri 2019-09-13 13:17:43 UTC; 1 day 16h ago
     Main PID: 1911 (code=exited, status=1/FAILURE)
    
    Sep 13 13:17:43 hostname systemd[1]: Starting ConfigServer Firewall & Security - csf...
    Sep 13 13:17:43 hostname csf[1911]: csf and lfd have been disabled, use 'csf -e' to enable
    Sep 13 13:17:43 hostname systemd[1]: csf.service: main process exited, code=exited, status=1/FAILURE
    Sep 13 13:17:43 hostname systemd[1]: Failed to start ConfigServer Firewall & Security - csf.
    Sep 13 13:17:43 hostname systemd[1]: Unit csf.service entered failed state.
    Sep 13 13:17:43 hostname systemd[1]: csf.service failed. 
    Code:
     perl /etc/csf/csftest.pl
    Testing ip_tables/iptable_filter...OK
    Testing ipt_LOG...OK
    Testing ipt_multiport/xt_multiport...OK
    Testing ipt_REJECT...OK
    Testing ipt_state/xt_state...OK
    Testing ipt_limit/xt_limit...OK
    Testing ipt_recent...OK
    Testing xt_connlimit...OK
    Testing ipt_owner/xt_owner...OK
    Testing iptable_nat/ipt_REDIRECT...OK
    Testing iptable_nat/ipt_DNAT...OK
    
    RESULT: csf should function on this server
     
     
  4. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    note that im completely locked out from creating cron in anyway as a root user
     
  5. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    how are you creating cronjob ?
    use crontab -l to view current cronjobs and crontab -e command to invoke nano text editor see guide at HowTo: Add Jobs To cron Under Linux or UNIX?

    you can't restart csf as you have it disabled, to reneable run
    Code (Text):
    csf -e
     
  6. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    cant enable csf it still says permission denied on cron

    also crontab i shows no active cronjobs

    crontab e shows empty
     
  7. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    you need to add cronjobs yourself via crontab -e command as it launches nano linux text editor where you add your cronjobs yourself - that is proper way of adding cronjobs

    how are you attempting to add cronjobs ? not via adding them to crontab -e launched nano linux editor ?

    post output from command
    Code (Text):
    csf -e
     
  8. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    i noticed from above output listing for /var/spool/cron, that you're missing root user's cron file at /var/spool/cron/root. Did you try editing /var/spool/cron/root directly ? that's a big no no, crontab -e is correct way to edit cronjobs for root user. So what you have done is probably accidentally delete /var/spool/cron/root file when you tried to incorrectly edit /var/spool/cron/root directly

    so re-create the file /var/spool/cron/root via commands below
    Code (Text):
    touch /var/spool/cron/root
    chmod 600 /var/spool/cron/root
    

    then using crontab -e command launch nano linux editor and re-add the default centmin mod cronjobs below and save and exit nano via ctrl+x and answer yes to save to exit - upon exiting crontab -e, your new cronjobs will register and be viewable via crontab -l command
    Code (Text):
    13 23 * * * /usr/local/src/centminmod/tools/autoprotect.sh >/dev/null 2>&1
    0 */4 * * * /usr/bin/cminfo_updater 2>/dev/null
    

    if you had other cronjobs setup properly, but are not lost when /var/spool/cron/root was deleted, then you can use cronjob log at /var/log/cron to get an idea what those cronjob commands were so to re-create them via crontab -e

    i.e. list last 100 lines of cronjob log at /var/log/cron
    Code (Text):
    tail -100 /var/log/cron

    if you had centmin.sh menu option 22 wordpress sites setup, there's a backup of their created cronjobs in files listed at /home/nginx/domains/${vhostname}/cronjobs/ where ${vhostname} is your wordpress domain name. Use crontab -e to re-add those backed up wordpress cronjobs too.
     
    Last edited: Sep 15, 2019
  9. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    when i tried csf -e i always get this error
    Error: Could not create /etc/cron.d/csf_update: Permission denied, at line 3702


    also no, i have not touched anything
    /var/spool/cron

    also i tried saving the file in

    crontab -e
    i get Permission denied, operation not permitted as a root user
     
  10. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    have you done this step first
     
  11. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    yes, i still get permission denied
     
  12. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    those steps to fix cronjob not csf so separate issues or could be related what is output from these commands
    inspecting line 3702 of /usr/sbin/csf which is called from running or creating /etc/cron.d/csf_update and filter grep on keyword 3702 with printing context of 5 lines before and after keyword match
    Code (Text):
    cat -n /usr/sbin/csf | grep -C5 3702

    also output to check if /etc/cron.d directory exists, you could have accidentally deleted that directory ?
    Code (Text):
    ls -lah /etc | grep cron
    ls -lah /etc/cron.d
    

    example of expected output on working system
    Code (Text):
    ls -lah /etc | grep cron
    -rw-------    1 root  root           541 Nov 20  2018 anacrontab
    drwxr-xr-x.   2 root  root          4.0K Sep  5 12:40 cron.d
    drwxr-xr-x.   2 root  root          4.0K Jul 16 07:00 cron.daily
    -rw-------    1 root  root             0 Nov 20  2018 cron.deny
    drwxr-xr-x.   2 root  root          4.0K Jan 18  2019 cron.hourly
    drwxr-xr-x.   2 root  root          4.0K Jun  9  2014 cron.monthly
    -rw-r--r--    1 root  root           466 Jan 18  2019 crontab
    drwxr-xr-x.   2 root  root          4.0K Jun  9  2014 cron.weekly
    

    Code (Text):
    ls -lah /etc/cron.d
    total 44K
    drwxr-xr-x.   2 root root 4.0K Sep  5 12:40 .
    drwxr-xr-x. 121 root root  12K Sep  5 12:40 ..
    -rw-r--r--    1 root root  128 Nov 20  2018 0hourly
    -rw-r--r--    1 root root   14 Feb  1  2013 csf-cron
    -rw-------    1 root root   46 Jan 18  2019 csf_update
    -rw-r--r--    1 root root   74 Aug 24 07:00 lfd-cron
    -rw-r--r--    1 root root  108 Oct 30  2018 raid-check
    -rw-------    1 root root  233 Sep  5 12:40 sysstat
    

    to double check if any commands you ran accidentally deleted files or directories you can inspect command history log filter egrep for words cron and rm and then pipe results into another egrep filter to return only august and september date timestamped entries for past 2 months
    Code (Text):
    history | egrep 'cron|rm ' | egrep '.09.19|.08.19'
    
     
  13. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    no i didnt accidently deleted anything

    auth.log is also completely empty
    ls -lah /etc | grep cron
    ls -lah /etc/cron.d

    both showed nothing

    etc/cron.d is completely empty
    i cant even create a test file inside as a root user

    also i noticed theres a new docker container that consumes 300% cpu resource and i never added that container.

    it leads to nginx.exe
    and on htop it showed nginx 300% despite nginx services are stopped..

    im suspecting that container is impersonating nginx
     
  14. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    nginx.exe is windows executable so doesn't belong on centos linux and if those are empty them someone deleted them and if you didn't install docker that that is worrying

    what about history output ?

     
  15. yunos

    yunos Member

    63
    3
    8
    Aug 8, 2015
    Ratings:
    +9
    Local Time:
    4:26 AM
    1.8.0
    history is completely empty, i did install docker myself though since i needed it to run bots on discord.

    i installed it from this guide: NaturalHistoryMuseum/scratchpads2
     
  16. eva2000

    eva2000 Administrator Staff Member

    41,719
    9,390
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +14,424
    Local Time:
    12:26 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    hmm could be sign of being hacked/compromised as the hacker would wipe history logs to hide traces of their commands etc but also can be case if you accidentally ran history -c command to clear history as well

    How long did you have csf firewall disabled for ? by disabling csf firewall you could of allowed a hacker to penetrate any open unprotected additional services you installed after installing centmin mod too. But hard to say what happened from the outside of your server from our perspective
    what is output for
    Code (Text):
    docker ps -a

    Code (Text):
    docker images

    Code (Text):
    docker info

    Code (Text):
    ps aufxw

    and to check when docker containers were started/created
    Code (Text):
    docker ps -a -q | while read id; do docker inspect --format='{{.Created}} {{.Config.Image}} {{.State.StartedAt}}' $id; done
     
  17. Rake-GH

    Rake-GH Premium Member Premium Member

    33
    19
    8
    Jul 29, 2019
    USA
    Ratings:
    +27
    Local Time:
    10:26 PM
    default
    default
    I got my money on this being a docker cryptominer, will be interesting to see how this plays out
     
    • Like Like x 1