Learn about Centmin Mod LEMP Stack today
Become a Member

Security Ouch VestaCP servers hacked !

Discussion in 'System Administration' started by eva2000, Apr 8, 2018.

Tags:
  1. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Some Centmin Mod users also use or have tried VestaCP in the past so this is just a heads up that alot of VestaCP users' servers have been reportedly hacked and compromised and sending out spam and DDOS attacking other sites from the compromised servers. Web hosts are suspending VestaCP servers. Full details and ongoing discussion at
    Worth keeping an eye on.

    From VestaCP forums, seem a command IP related to the hacker is 119.82.29.17. Centmin Mod users might want to block that IP via CSF Firewall - labeled with comment = badvestacp
    Code (Text):
    csf -d 119.82.29.17 badvestacp
    

    example output after running command
    Code (Text):
    csf -d 119.82.29.17 badvestacp
    Adding 119.82.29.17 to csf.deny and iptables DROP...
    csf: IPSET adding [119.82.29.17] to set [chain_DENY]
    

    CSF Firewall grep IP check that it's blocked
    Code (Text):
    csf -g 119.82.29.17
    
    Table  Chain            num   pkts bytes target     prot opt in     out     source               destination        
    No matches found for 119.82.29.17 in iptables
    
    IPSET: Set:chain_DENY Match:119.82.29.17 Setting: File:/etc/csf/csf.deny
    
    ip6tables:
    
    Table  Chain            num   pkts bytes target     prot opt in     out     source               destination        
    No matches found for 119.82.29.17 in ip6tables
    
    csf.deny: 119.82.29.17 # badvestacp - Sun Apr  8 12:43:33 2018
    

    Centmin Mod 123.09beta01 users might want to install and setup optional Auditd logging via tools/auditd.sh outlined at https://community.centminmod.com/th...td-support-added-in-latest-123-09beta01.9071/. This allows you to somewhat track directory and file access and write changes for common Centmin Mod and CentOS system and vhost files. Dedicated tools/auditd.sh discussion thread is here.
     
    • Informative Informative x 3
    • Like Like x 2
  2. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Looks like it might be a Linux/Xor.DDoS (also known as DDoS.XOR or Xorddos) malware. Very old 2015 blog article on this at https://bartblaze.blogspot.com.au/2015/09/notes-on-linuxxorddos.html

    Centmin Mod 123.09beta01 users can use just updated cminfo netstat command to get an overview of netstat incoming and outgoing connections and use auditd's authenticated user reports. Example of using tools/auditd.sh to track sudo users

    From https://forum.vestacp.com/viewtopic.php?f=10&t=16556#p68515
     
    • Informative Informative x 2
    • Funny Funny x 1
  3. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Ouch compromised VestaCP servers are launching DDOS attacks from DigitalOcean droplets which have been hacked https://status.digitalocean.com/incidents/jzszyktwsrss
    Looks like DO blocking 8083 port will effectively prevent any VestaCP access regardless of if the VestaCP install instance is compromised !

    https://www.digitalocean.com/commun...-of-vestacp-vulnerability-from-april-8th-2018

     
    • Informative Informative x 1
  4. ArisC

    ArisC Active Member

    113
    27
    28
    Jun 1, 2017
    Ratings:
    +52
    Local Time:
    12:57 PM
    Nginx Latest
    MariaDB Latest
    It was about to happen something....
     
  5. Revenge

    Revenge Active Member

    408
    85
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +302
    Local Time:
    10:57 AM
    1.9.x
    10.1.x
    Vesta released a new version with the exploit fixed.
    Nevertheless, only allow access to port 8083 for your own IP address.
     
  6. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Folks might want to continue reading at Got 10 VestaCP servers exploited - Vesta Control Panel - Forum as one person reported they still got compromised even when they started out with their port changed out of box apparently. Or maybe they did port change after being infected ? Still if compromised best to backup all data, reinstall OS and restore backed up data. That's usual way.
     
  7. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    More from https://www.digitalocean.com/commun...-of-vestacp-vulnerability-from-april-8th-2018

     
  8. Revenge

    Revenge Active Member

    408
    85
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +302
    Local Time:
    10:57 AM
    1.9.x
    10.1.x
    It's a bit strange because the only way to know if a server have Vesta installed is to ping the port 8083. If the user changed the port, and don't imagine the attacker to scan all ports in all ip's.

    What i imagine the attacker did was to scan the ip range of Digital Ocean and others like OVH etc, for port 8083. When found one open, it simple used the exploit to access the Vesta Panel.

    I always say this to all the people i know that manage a server. Block all ports(minus 80/443 of course) in your server and only open the ones you need for your own IP address or the one you need.

    Not much time ago we saw what happened with Memcached, and now with Vesta. This kind of things will continue to happen because people don't care if they have their port's open or not.

    I trully recommend your script because i know you care about security.
     
    • Like Like x 1
    • Agree Agree x 1
  9. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yeah it is strange unless they changed to something like 8089, then hacker could of scanned 8083, 8084 all the way to 8089 to check. Hence why I am following the VestaCP exploit to see how much of it is relevant for Centmin Mod if any and/or if the exploit is something other than direct VestaCP code. Interesting to learn that before VestaCP fixed updates, logging was disabled effectively by piping log data to /dev/null so there's no logs to troubleshoot stuff - there would be no way to investigate i.e. nginx logs ? They seem to have changed that to enable logging now though.

    As DigitalOcean link stated, looks like VestaCP pulled their repo packages too Got 10 VestaCP servers exploited - Page 31 - Vesta Control Panel - Forum so maybe they're investigating if their VestaCP repo has been compromised upstream ? That would be one explanation as to why even if ports were changed, VestaCP folks still got hacked.

    +1 Pretty much what Centmin Mod default installed CSF Firewall is there for whitelisting some required ports and blocking everything else. CSF Firewall & Centmin Mod also detects user's IP address at time of install to whitelist that IP address too. CSF Firewall also supports whitelisting dynamic ISP ip addresses as outlined here.

    Indeed I have to since I also use Centmin Mod extensively myself :D

    Though this VestaCP situation is the main reason why Centmin Mod is non-gui based for pure SSH shell based menu. Adding a gui control panel that is public web facing introduces additional attack vectors for potential compromise. Not familiar with VestaCP API other than Vesta Control Panel — Documentation but seems like that is also another avenue for potential compromise i.e. how are VestaCP users safeguarding and storing all those API related scripts they write up or use ?
     
  10. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Thinking back to past experience in cleaning up hacked servers for clients, sometimes there's more than one infection or entry point. Hackers like to leave backdoors so they can re-enter a compromised server as well. I've seen compromised servers where the hacker made and configured multiple copies of the compromised scripts in public web accessible directories that were several directories deep !

    The initial compromise could of been months ago to setup backdoors first.

    Depending on how to check for old or modified files, it can be easy to fool some tools to see which files were changed

    i.e. with touch

    Code (Text):
    stat index.php
      File: ‘index.php’
      Size: 0               Blocks: 0          IO Block: 4096   regular empty file
    Device: fd01h/64769d    Inode: 81802289    Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2018-04-09 12:35:23.132433155 +0000
    Modify: 2018-04-09 12:35:23.132433155 +0000
    Change: 2018-04-09 12:45:23.131878840 +0000
     Birth: -
    

    Modify file so it's 3 days older timestamp
    Code (Text):
    touch -d "3 days ago" index.php
    

    Code (Text):
    stat index.php
      File: ‘index.php’
      Size: 0               Blocks: 0          IO Block: 4096   regular empty file
    Device: fd01h/64769d    Inode: 81802289    Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2018-04-06 12:52:01.984318757 +0000
    Modify: 2018-04-06 12:52:01.984318757 +0000
    Change: 2018-04-09 12:52:01.983577779 +0000
     Birth: -
    

    Find is empty for less than a day old files
    Code (Text):
    find . -mtime -1             
    .
    

    Find returns index.php for older than a day files
    Code (Text):
    find . -mtime +1             
    ./index.php
    

    though changing to -ctime will pick it up
    Code (Text):
    find . -ctime -1
    .
    ./index.php
    
     
    • Informative Informative x 1
  11. Revenge

    Revenge Active Member

    408
    85
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +302
    Local Time:
    10:57 AM
    1.9.x
    10.1.x
    There is new information. The only servers hacked, were servers where VestaCP was installed in the last few days. This means their repositories were hacked.

    This explains why some people with a different port configured and/or the port blocked, were still hacked anyway.

     
    • Informative Informative x 2
  12. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Actually that could make sense especially if VestaCP repo was managed by VestaCP install. The hacker could of got in due to the password flaw to the VestaCP repo first and compromised the repo files.
     
  13. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Strange indeed Got 10 VestaCP servers exploited - Page 39 - Vesta Control Panel - Forum. This is after latest VestaCP updates I believe
     
  14. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Seems like this could be an expensive issue for some VestaCP users. I recall some web hosts terms and conditions have clauses that DDOS attacks are billable with penalties for the users so potentially web hosts could charge and bill VestaCP users for the outgoing DDOS attack costs/bandwidth :eek: Other web hosts have acceptable usage policies which would allow them to suspend or cancel users' servers/accounts too !
     
  15. Revenge

    Revenge Active Member

    408
    85
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +302
    Local Time:
    10:57 AM
    1.9.x
    10.1.x
    Reading the vesta topic, it seems there are many misinformation. I don't even know if there are people with that intention.

    Vesta team confirmed their repo was not compromised.

    For now, it seems the exploit was this code found by a user:

    VestaCP fixed that here:

    https://github.com/serghey-rodin/vesta/commit/3fdee2975db0c80419a0dfefff3c10a2c4de6410
     
  16. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yeah just read that at https://forum.vestacp.com/viewtopic.php?f=25&p=69296#p69296. Misinformation of course is because no one 100% knows where the attack vector(s) come from.
    But that doesn't explain folks reportedly being compromised on fresh installs ? Though it's confusing as compromised folks aren't exactly explaining the state of the VestaCP installs whether it's fresh install or just updating an existing compromised install.

    Then there was VestaCP installs totally locked down and hidden that got compromised too.

    Maybe instead of default port 8083, installs should just randomise the port used so every install would have a different unique port ?
     
  17. Revenge

    Revenge Active Member

    408
    85
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +302
    Local Time:
    10:57 AM
    1.9.x
    10.1.x
    For now there are no new servers hacked, at least no one is complaining after the VestaCP update. The guy that exploited the servers to ddos a Chinese IP Address, would continue for sure to hack as most servers as possible, since the objective is for ddos.

    If we look well for the thread, only 1 or 2 users said their servers were hacked with the port 8083 blocked. If their repo was indeed not compromised, i find it almost impossible to hack a server through a service that have the only port that uses blocked.
    I only see here 2 possible reasons, the user thinks the port is closed but it is not, or the user just want to cause panic because its funny for him.

    PS: The guy that received the OVH email, was probably already hacked before the update but didn't noticed.
     
  18. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yeah that could be a possibility. Only time will tell really
     
  19. Revenge

    Revenge Active Member

    408
    85
    28
    Feb 21, 2016
    Portugal
    Ratings:
    +302
    Local Time:
    10:57 AM
    1.9.x
    10.1.x
    Vesta Team explanation:

     
    • Informative Informative x 3
  20. eva2000

    eva2000 Administrator Staff Member

    36,054
    7,910
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,192
    Local Time:
    7:57 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    thanks @Revenge for the info - interesting method there. Can't get to the forums right now though

    upload_2018-4-11_4-1-36.png
     
..