Learn about Centmin Mod LEMP Stack today
Register Now

Installing an email server (Listmonk, PostgreSQL) on a new vhost, in addition to a forum (Xenforo)

Discussion in 'Other Web Apps usage' started by MaximilianKohler, Oct 10, 2023.

  1. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    I don't think that has anything to do with it. I only temporarily changed the file before reverting back, and the other settings I listed before that one worked fine. I just double checked as well.


    Per my previous comment, the html vhost page also loads fine. There's some issue with docker & nginx.

    I had another read over the vhost setup https://centminmod.com/nginx_domain_dns_setup.html to see if I could spot something there that may be causing a conflict. This is the only thing that stands out to me:
    I did test removing that line before from the nginx config, but decided to give it another shot since perhaps that contributes to the "broken page" I got to before. And I see there's two of them. I tried removing them but it doesn't fix the broken page.

    Removing those lines changes the nginx error logs from
    Code:
    [error] 9797#9797: *1 open() "/home/nginx/domains/test1.example.com/public/public/custom.js" failed (2: No such file or directory), client: <myip>, server: test1.example.com, request: "GET /public/custom.js HTTP/2.0", host: "test1.example.com", referrer: "https://test1.example.com/"
    to
    Code:
    [error] 7563#7563: *11 open() "/usr/local/nginx/html/public/custom.js" failed (2: No such file or directory), client: <myip>, server: test1.example.com, request: "GET /public/custom.js HTTP/2.0", host: "test1.example.com", referrer: "https://test1.example.com/"
    Maybe that's another clue, but I don't know what to do with that info.

    Isn't the fact that I can't curl these IPs a problem?
    Code:
    curl 127.0.0.1:5870
    curl: (56) Recv failure: Connection reset by peer
    
    curl 0.0.0.0:5870
    curl: (56) Recv failure: Connection reset by peer
    
    curl 127.0.0.1:9000
    curl: (56) Recv failure: Connection reset by peer
    Could it be a firewall issue? How would I check that? I found suggestions to check "from inside the docker container". I ran:
    Code:
    sudo ip addr show docker0
    4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
        link/ether 02:42:a9:60:8e:f4 brd ff:ff:ff:ff:ff:ff
        inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
           valid_lft forever preferred_lft forever
    
    curl 172.17.0.1:5870
    And it output the listmonk html page.

    Solution:
    I found the command "csf -x" to disable the firewall and that worked.

    Since "docker ps" shows "0.0.0.0:5870->9000/tcp" I edited "/etc/csf/csf.conf" per https://centminmod.com/csf_firewall.html to add "5870" to "TCP_OUT" then "TCP_IN". That didn't work, but "5870:9000" or just "9000" to "TCP_OUT" did. I'm wondering if that's going to cause an issue since php-fpm uses that port. Per this port guide, it sounds like it's not a problem.

    Strange that I have not allowed 5870 in the csf.conf, only 9000, yet "curl 0.0.0.0:5870" and "curl 127.0.0.1:5870" now works, but "curl 0.0.0.0:9000" and "curl 127.0.0.1:9000" still show "Recv failure: Connection reset by peer".

    It looks like 5432 also has to be allowed for the postgres db, but adding "5432" to TCP_OUT and TCP_IN didn't work. Since "docker ps" shows "0.0.0.0:9432->5432/tcp" I also tried "5432:9432" and "5432,9432", which didn't work, but "9432:5432" to TCP_OUT worked. I wonder then if I should be using "5870:9000" instead of "9000"?

    According to a search, "5870:9000" opens all ports between 5870 and 9000, so it would be better to only open 9000. Strange then that "5432:9432" doesn't work but "9432:5432" does.

    BTW, it would be nice if I could edit my OP to include the solution so that people don't have to read through the entire thread to find it.
     
  2. eva2000

    eva2000 Administrator Staff Member

    51,696
    11,941
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,432
    Local Time:
    10:52 PM
    Nginx 1.25.x
    MariaDB 10.x
    The semi colon in CSF Firewall is a range list so I wouldn't do that that will open alot of ports not recommended

    That is because TCP in Port 9000 is used by php-fpm service and you shouldn't open that port in CSF Firewall for security reasons
     
  3. eva2000

    eva2000 Administrator Staff Member

    51,696
    11,941
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,432
    Local Time:
    10:52 PM
    Nginx 1.25.x
    MariaDB 10.x
    Disabling CSF Firewall isn't recommended as it isn't really opening all ports but rather disabling all firewall rules that have been configured to be properly allowed. So inadvertently all you do is make a mess of whitelisted ports that Centmin Mod CSF Firewall configured to be allowed for software that needs to run properly
     
  4. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    Isn't disabling the firewall completely reversible? I did it as a temporary measure to check if the problem was the firewall. From your last comment it sounds like disabling the firewall permanently changes things?

    I asked on the listmonk github what exact ports need to be opened. We'll see what they say. Based on your response it then sounds like "9432:5432" is opening all ports greater than 9432 and less than 5432? IE: all ports NOT between 5432 and 9432, which is what "5432:9432" would do?
     
  5. eva2000

    eva2000 Administrator Staff Member

    51,696
    11,941
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,432
    Local Time:
    10:52 PM
    Nginx 1.25.x
    MariaDB 10.x
    If you use csf -x to disable CSF Firewall, you need to restart (csf -ra) or enable it again (csf -e) for CSF Firewall to work - it won't automatically re-enable itself.

    If linkmonk's PostgreSQL runs on 127.0.0.1:9432 then you won't need to do anything in CSF Firewall as internal ports aren't blocked for stuff listening locally on 127.0.0.1 usually. But depends on how you installed and configured Docker as well.
     
  6. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    In listmonk I get a bunch of "error doing x: dial tcp 172.18.0.2:5432: connect: connection timed out" which were only solved by adding "9432:5432" to the TCP_OUT rules.

    I did more testing and it seems that "9432:5432" works because it's an invalid port range, and thus it's disabling something.
    Code:
    Error: FASTSTART: (TCP_OUT IPv4) [] [iptables-restore v1.4.21: invalid portrange (min > max)]. Try restarting csf with FASTSTART disabled, at line 5790
    
    I don't know what it's disabling (and thus what I need to fix), because even a port range of 1:60000 doesn't work to stop the error. Where do I start to debug this?

    According to https://jsherz.com/docker/configser...iguring-configserver-firewall-for-docker.html docker already automatically configures the firewall to allow it to access what it needs.

    https://serverok.in/docker-csf-firewall didn't work.

    Interesting... "service docker restart" worked... but they say it's only a temporary fix, and I confirmed that restarting CSF "unfixes it". Perhaps it will work in conjunction with ETH_DEVICE_SKIP = "docker0"? Nope, restarting CSF still kills it.

    I found this and set DOCKER = "1" in the CSF config, but that didn't fix it either. I see that the docker range is "172.17.0.0/16" but the error is "dial tcp 172.18.0.2:5432". I guess 172.18.0.2 is a postgres IP range, but there are no results for "postgres" or "172.18" in the CSF config.

    Nevermind, it's not postgres, it's listmonk_db. Per https://discuss.elastic.co/t/error-...rror-dial-tcp-172-18-0-2-i-o-timeout/306411/3 I found the "docker network inspect" command.
    Code:
    docker network ls
    NETWORK ID     NAME                DRIVER    SCOPE
    47f5b46070d0   bridge              bridge    local
    2f1937b81c6c   host                host      local
    248eba388dfb   listmonk_listmonk   bridge    local
    632a3a18dd6d   none                null      local
    
    docker network inspect listmonk_listmonk
    
    listmonk_listmonk
    172.18.0.0
    172.18.0.1
    
    listmonk_app
    172.18.0.3/16
    
    listmonk_db
    172.18.0.2/16
    Per https://serverfault.com/questions/1020400/running-docker-containers-only-local-behind-csf-firewall I tried
    Code:
    ETH_DEVICE_SKIP = "docker0,listmonk_listmonk,listmonk_app,listmonk_db"
    But that didn't work, even in conjunction with DOCKER = "1".

    I see in the csf.conf it says
    There is no "csfpost.sh" file, and https://centminmod.com/csf_firewall.html has no mention of "csfpost.sh", so I decided to try adding all the IPs to csf.ignore because csf.allow references that file. It didn't work. So I added them to csf.allow as well and it still didn't work.

    Per https://centminmod.com/csf_firewall.html I tried:
    Code:
    csf -a 172.18.0.0 listmonk
    csf -a 172.18.0.1 listmonk
    csf -a 172.18.0.3/16 listmonk
    csf -a 172.18.0.2/16 listmonk
    
    csf -a 172.18.0.0 listmonk
    Adding 172.18.0.0 to csf.allow and iptables ACCEPT...
    csf: IPSET adding [172.18.0.0] to set [chain_ALLOW]
    csf -a 172.18.0.1 listmonk
    add failed: 172.18.0.1 is one of this servers addresses!
    csf -a 172.18.0.3/16 listmonk
    Adding 172.18.0.3/16 to csf.allow and iptables ACCEPT...
    csf: IPSET adding [172.18.0.3/16] to set [chain_ALLOW]
    csf -a 172.18.0.2/16 listmonk
    Adding 172.18.0.2/16 to csf.allow and iptables ACCEPT...
    csf: IPSET adding [172.18.0.2/16] to set [chain_ALLOW]
    
    That added them to csf.allow again (I removed them first), but it didn't fix the issue...

    I additionally added "5432" to TCP_IN and TCP_OUT and that still didn't fix it.

    @eva2000 you use docker so you've dealt with this before right? Per the documentation available I think I've tried everything reasonable. I think that page could also use some updating. It doesn't even mention the docker option in the csf.conf file.
     
    Last edited: Nov 29, 2023
  7. eva2000

    eva2000 Administrator Staff Member

    51,696
    11,941
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,432
    Local Time:
    10:52 PM
    Nginx 1.25.x
    MariaDB 10.x
    Docker is left to end users to figure out as I wouldn't provide free support for it. If folks want to hire me to configure it for their specific docker based web application that's another story :)

    My usage case for Docker on Centmin Mod hasn't required public access only local internal access. Generally, you don't want to expose Docker container images/services to the public internet directly. So haven't had to do much in the way of the CSF Firewall configuration besides adjusting the TCP outbound port allowed list if the app communicates outbound to ports other than already allowed TCP port outbound 80 and 443.

    I whipped up something quickly you can try https://gist.github.com/centminmod/16eb4b9ad69783cfc4984262ab0e44a3 but rest you'd have to figure out. As I wrote in the gist, generally you don't want to give direct public internet access to Docker containers by opening CSF Firewall TCP inbound ports.
     
  8. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    Thanks! Unfortunately that didn't work:
    Code:
    docker network inspect bridge | jq -r '.[].IPAM.Config[].Subnet'
    172.17.0.0/16
    docker network inspect listmonk_listmonk | jq -r '.[].IPAM.Config[].Subnet'
    172.18.0.0/16
    
    172.17.0.0/16 # docker
    172.18.0.0/16 # listmonk
    
    iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
    iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE
    
    So I tried running "sudo systemctl daemon-reload" per https://jsherz.com/docker/configser...iguring-configserver-firewall-for-docker.html, and I'm back to a 502 bad gateway.

    I ran "csf -x", 502 error gone. I deleted the /etc/docker/daemon.json file and ran the reset commands again
    Code:
    sudo systemctl daemon-reload
    sudo systemctl restart docker
    sudo csf -e
    
    And it's working. I noticed this error in the output:
    Code:
    Running /etc/csf/csfpost.sh
    sh: /etc/csf/csfpost.sh: /bin/sh^M: bad interpreter: No such file or directory
    
    which can be fixed by changing the invisible line break characters.

    I fixed the file and ran "csf -ra" and listmonk is back to being broken. I redid everything and added all of these ports (9432,5432,5870,33318,46525) to TCP and TCP6 IN and OUT:
    Code:
    netstat -plant | grep docker
    tcp        0      0 0.0.0.0:5870            0.0.0.0:*               LISTEN      12679/docker-proxy
    tcp        0      0 0.0.0.0:9432            0.0.0.0:*               LISTEN      12424/docker-proxy
    tcp        0      0 172.18.0.1:33318        172.18.0.2:9000         ESTABLISHED 12679/docker-proxy
    tcp        0      0 127.0.0.1:5870          127.0.0.1:46526         ESTABLISHED 12679/docker-proxy
    tcp6       0      0 :::5870                 :::*                    LISTEN      12684/docker-proxy
    tcp6       0      0 :::9432                 :::*                    LISTEN      12429/docker-proxy
    And it went back to 502 bad gateway. I deleted the /etc/docker/daemon.json file again and ran the reset commands again, and 502 error is gone but listmonk is still broken. So:
    • The /etc/docker/daemon.json file causes a 502 bad gateway.
    • Adding ports (9432,5432,5870,33318,46525) to TCP and TCP6 IN and OUT (plus port 9000 to TCP_OUT and TCP6_OUT) is insufficient.
    • Adding the docker bridge and listmonk_listmonk IP ranges to csf.allow and csfpost.sh is insufficient.

    The lack of info on the configserver.com forums and knowledgebase doesn't give me much confidence that CSF is a great firewall option. Docker is super popular and they have zero information on how to allow it past their firewall.
     
  9. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
  10. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    I was looking up a guide that installs listmonk without docker and came across this Listmonk with Docker on AlmaLinux 8 guide. They do nothing but allow port 9000 on TCP, so that made me wonder what firewall AlmaLinux has.
    And docker integrates seamlessly with firewalld https://docs.docker.com/network/packet-filtering-firewalls/#integration-with-firewalld. It might be too lenient by default but it's possible to configure it to be more strict https://stackoverflow.com/questions/66527143/how-to-configure-firewalld-with-docker-20-10.

    Eva, do you plan to stick with CSF even with AlmaLinux? Or will you switch to firewalld?
     
  11. eva2000

    eva2000 Administrator Staff Member

    51,696
    11,941
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,432
    Local Time:
    10:52 PM
    Nginx 1.25.x
    MariaDB 10.x
    How are you editing files on the server, this is indicative of not editing files properly, which probably be a persistent issue you carry through for all files you edit on server!

    Easiest way to edit configuration or any files on your server is via logging into your server via ssh and directly editing them using nano or vim linux text editors.

    For nano which you can read up more about nano here and here. For vim text editor read here and here and thread at WebPerf - Popular Vim Commands – a Comprehensive Vim Cheatsheet

    Also there's numerous online how to use guides for nano and vim you can search for via google :)
     
  12. eva2000

    eva2000 Administrator Staff Member

    51,696
    11,941
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,432
    Local Time:
    10:52 PM
    Nginx 1.25.x
    MariaDB 10.x
    CSF Firewall is the one unless it breaks and haven't had issues with my Docker usage :)

    IIRC, I've used CSF Firewall for 2 decades now :D
     
  13. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    Generally via FileZilla & notepad++. I find notepad++ to be much better than nano. The issue I ran into seems to only be a problem if I create the file on my PC and then upload it, or by "create new file" in FileZilla.

    Configserver (CSF Firewall) haven't even approved my comments on their forum for well over a day, so I have a pretty poor opinion of them right now. If there is a way to switch to firewalld with the AlmaLinux update I'd definitely take that. Otherwise, I'll just try to avoid using docker with centmin mod. Or maybe I'll look up a guide to uninstall CSF and install firewalld.

    For now, I found another way around this issue:
    Install listmonk without docker:
    I tried it and it works. https://listmonk.app/docs/installation/#binary
    Code:
    uname -m
    x86_64
    cat /proc/cpuinfo
    
    Told me I had an AMD CPU. So I downloaded the linux_AMD64 binary.
    Code:
    cd /home/nginx/domains/test1.example.com/public
    
    Uploaded the listmonk files there via FileZilla.
    Grant execute permissions for the listmonk binary to owner -- 744 via FileZilla, or:
    Code:
    chmod +x listmonk
    Then:
    Code:
    ./listmonk --new-config
    Edited config.toml to change the port to 9011. I also had to change the postgres port from 5432 to 9432.
    I didn't have to run
    Code:
    ./listmonk --install
    because that seems to only setup postgres, and my previous docker install already did that. So I just ran:
    Code:
    ./listmonk
    2023/11/30 20:04:55 main.go:102: v2.5.1 (a6a2b69 2023-08-15T15:49:28Z, linux/amd64)
    2023/11/30 20:04:55 init.go:145: reading config: config.toml
    2023/11/30 20:04:55 init.go:273: connecting to db: localhost:9432/listmonk
    2023/11/30 20:04:55 init.go:593: media upload provider: filesystem
    2023/11/30 20:04:55 init.go:517: loaded email (SMTP) messenger: username@smtp.yoursite.com
    ⇨ http server started on 127.0.0.1:9011
    
    and changed proxy_pass to "proxy_pass http://127.0.0.1:9011;". And it's up and running at test1.example.com. I didn't even have to add port 9011 to the firewall.

    But to keep it running you have to set it up as a service: https://listmonk.app/docs/installation/#tutorials
     
  14. eva2000

    eva2000 Administrator Staff Member

    51,696
    11,941
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,432
    Local Time:
    10:52 PM
    Nginx 1.25.x
    MariaDB 10.x
    If you edit on PC and upload, you can inadvertently break Centmin Mod software config files and operations if Centmin Mod and software can't properly read the contents of the config files etc. Did you edit docker files and other nginx vhost/CSF Firewall the same way via PC edit and FTP upload? That could of been partially why you had problems. If you do upload edited files from PC, run file through dos2unix command to possible fix them
    Code (Text):
    dos2unix filename.conf

    If you don't know how to configure firewalld for Centmin Mod software, I'd advise against that as uninstalling CSF Firewall will undo all specific rules setup for Centmin Mod software to operate properly with. Just install/switch to firewalld, will not configure Centmin Mod software properly.
    Glad to hear.
     
  15. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    I generally use this method for everything. Nothing bad happens if I'm just editing files. It only seems to happen when creating a new file. I submitted a bug report, but it can also be avoided by using "nano" for creating files, or manually setting the line break to Unix LF with notepad++.

    It's possible this issue caused the 502 error with the daemon.json file, but since that file is only for preventing docker from creating iptables rules, it wouldn't have fixed my problem.
     
  16. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    For peace of mind, in case I was wrong about what /etc/docker/daemon.json does, I decided to redo the steps making sure that I created the daemon.json file with the correct line breaks. It seems my understanding was correct. The daemon.jason file prevents docker from whitelisting the ports/IPs so "service docker restart" no longer temporarily fixes things.

    First it was on a 502 error, then I ran csf -ra multiple times over a period of time and the 502 eventually went away, but the "error fetching lists: dial tcp 172.18.0.2:5432: connect: connection timed out" still occurs.

    Indeed, some strange behavior. I added more ports to csf.conf, ran "systemctl daemon-reload", no change. Ran "systemctl restart docker", now 502. Ran "csf -ra" multiple times, still 502.

    EDIT: I also confirmed that this script https://forum.configserver.com/viewtopic.php?p=28680#p28680 works by itself, without the need for the other two configurations he mentions. I don't even have an "/etc/sysconfig/docker", and I don't know what to do with the "open_port" script he shows. And the script also works in conjunction with eva's /etc/docker/daemon.json file.
     
    Last edited: Dec 2, 2023
  17. MaximilianKohler

    MaximilianKohler Member

    150
    1
    18
    Jun 23, 2023
    Ratings:
    +13
    Local Time:
    5:52 AM
    In the nginx file, one or more of these cause the listmonk homepage to not load correctly so I commented them out:

    # include /usr/local/nginx/conf/php.conf;
    # include /usr/local/nginx/conf/pre-staticfiles-local-test1.domain.com.conf;
    # include /usr/local/nginx/conf/pre-staticfiles-global.conf;
    # include /usr/local/nginx/conf/staticfiles.conf;
    # include /usr/local/nginx/conf/drop.conf;
    #include /usr/local/nginx/conf/errorpage.conf;
    # include /usr/local/nginx/conf/vts_server.conf;