Want to subscribe to topics you're interested in?
Become a Member

Email Steps to ensure your site/server email doesn't end up in spam inboxes

Discussion in 'Domains, DNS, Email & SSL Certificates' started by eva2000, Apr 10, 2016.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    53,531
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    11:08 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    To ensure your site/server sent emails do not end up in users' spam inboxes, you need proper SPF/TXT, DKIM, DMARC and reverse PTR DNS records setup for your domain and server's main hostname (setup via Getting Started Guide Step 1) as outlined below.

    There is a distinction between your site domain and server's main hostname:
    1. server's main hostname - If you use a web app like Wordpress, Xenforo, IP.Board, Magento, etc to sent emails. Then emails can either be sent as is via Centmin Mod Postfix MTA server via root@host.domain.com where receiving email servers would evaluate the SPF, DKIM, DMARC and PTR DNS records for sending domain @host.domain.com. If server's main hostname i.e. host.domain.com has valid DNS records, then email will be less likely sent to spam box or being rejected by receiving email server. If you use Cloudflare, DO NOT enable orange clooud proxy on server's main hostname DNS record as you prevent receiving mail servers looking up the main hostname's real IP address to verify your mail server.
    2. your domain - If your web app allows you to set a from email field and you sent emails from say user@domain.com, then receiving email servers would evaluate the SPF, DKIM, DMARC and PTR DNS records for sending domain @domain.com so your domain.com would need to have those DNS records setup.
    Centmin Mod is provide as is, so short of scripted related bugs or issues, any further optimisation to the web stack components - nginx, php-fpm, mariadb mysql, csf firewall etc or web app specific configurations are left to the Centmin Mod user to deal with. So I do not provide any free support for such things as SPF/TXT and DNS records setup. For SPF/DNS, you're best to ask your web host and/or domain registrar and follow below instructions and check out thread at Email - error in PTR, mails goes to gmail Spam folder.

    However, Centmin Mod users are free to help each other out and ask questions or give answers on this community forum. My hopes are that this community forum evolves so that more veteran long time Centmin Mod users help new Centmin Mod users out :)

    Generally, to ensure email sent from your Centmin Mod LEMP server's postfix MTA doesn't end up in email recipient's spam box, you need to at least have proper DNS level configurations for
    1. SPF / TXT record setup in DNS with your DNS provider. SPF / TXT record is a DNS text record so how you add it will depend on your DNS provider. Some info here, here and here. Also SPF guides from some Domain/DNS providers such as Namecheap.com, Godaddy.com, and check out SPF Wizard - SPF Generator DNS tool. Ensure both your site's domain name and server's main hostname from Getting Started Guide step 1 have valid SPF TXT records
    2. Reverse PTR record, you will need to ask your web host for that. Some web hosts and VPS providers use SolusVM control panels and allow you to setup a PTR record yourself. Basically it points your server's IP to your domain name. So when a reverse lookup on the IP is made, it resolves to your domain name. For instance the PTR record for this forum's IP resolves to community.centminmod.com when you do a reverse IP lookup at IP Address Lookup (IPv4 & IPv6). I use Linode.com for my forum hosting and their control panel has the relevant PTR record support for reverse dns.
    3. DKIM setup using addons/opendkim.sh addon script for Centmin Mod users' email sent from their @mainhost.yourdomain.com. For emails @yourdomain.com, your 3rd party email provider will have DKIM setup instructions.
    4. DMARC record setup. You can use Postmark's DMARC setup service for this. More info on DMARC.
    5. Google Postmaster Tools. Sign up your site domains for this service. Postmaster Tools FAQ
    6. Ensuring server's native Postfix MTA mailserver's outbound SMTP port is working as some VPS providers now block them by default. So you may need to setup Postfix SMTP relays with a 3rd party SMTP provider. For example, DigitalOcean SMTP Outbound Email Blocked By Default

    IP Blacklist Check



    Also check if your web site and server IP addresses are in spam blacklists using services like:
    Straight from Amazon SES best practices guide, you should also check the URL links that you use in your emails against blacklists as well, because some ISPs will block email with blacklisted links. URIBL.com and SURBL.org are two very useful sites that you can use to determine whether your links are blacklisted. Remember to check any links provided to you by a third party or any link shorteners, which obfuscate your domain or links.

    Transactional Email Providers



    For forum, CMS etc outbound email and best email deliverability, best to use a 3rd party SMTP transactional email provider. Discussion of provider options can be read at Email - Mandrill users will be required to have a paid monthly MailChimp account. I use Amazon SES for this forum's emails.

    SPF Setup Guides


    SPF Wizards



    You can use SPF wizards to try and get the correct settings

    SPF Testing Tools


    DMARC Testing Tools


    DKIM Testing Tools


    Email Deliverability Testing Tools


    • Email Deliverability Tool - Check Your Deliverability Report - MxToolBox - to test against server sent email send test email from server via SSH
      Code (Text):
      echo "test" | mail -s "test" ping@tools.mxtoolbox.com
      and then on web page to search for your email which would be root@host.yourdomain.com where it would be your server's main hostname reported via command
      Code (Text):
      hostname -f
    You would get a report similar to one below returned from the search

    upload_2017-7-28_5-53-37.png


    Microsoft & Outlook Email


     
    Last edited: Jul 28, 2022
  2. eva2000

    eva2000 Administrator Staff Member

    53,531
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    11:08 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  3. eva2000

    eva2000 Administrator Staff Member

    53,531
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    11:08 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Added 5th step to sign up your domains for Google Postmaster Tools to monitor your domains spam status reported by users who receive your emails.

    i.e. for centminmod.com for past 120 days

    postmaster-tools-1.png postmaster-tools-4.png

    postmaster-tools-240418-01.png
     
    Last edited: Oct 8, 2016
  4. eva2000

    eva2000 Administrator Staff Member

    53,531
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    11:08 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    To elaborate more on 1st post in this thread, more info provided below.

    Troubleshooting Proper Email Sending



    To troubleshoot and/or ensure server sent emails via Postfix and from @domain.com your domains are delivered to their destination, do the following:

    Ensure Proper DNS Records Setup



    Ensure you have setup proper SPF, DKIM, DMARC & PTR DNS records for your main hostname i.e. hostname.domain.com and all @domain.com domain names as per outlined and linked instructions at here and Getting Started Guide Step 1 for main hostname (i.e. hostname.domain.com). The addons/opendkim.sh addon in 123.09beta01 will help you setup SPF/DKIM for main hostname of your Centmin Mod server. Without these DNS records for your main hostname and all domain names, the likelihood of your server sent and @domain.com sent emails delivering to target inboxes is lessened and may end up in user's spam box or being rejected.

    You can use these online DNS toolboxes to verify you have properly setup SPF, DKIM, DMARC and PTR DNS records

    Testing Server Sent Emails



    Use Newsletters spam test by mail-tester.com to send a test email from your server to the email address listed on the mail-tester.com web site - DO NOT leave the site yet though.

    To Test Server Sent Email

    For example if the site lists email web-1a0xv@mail-tester.com, in SSH you use mail command line to sent a test email with subject "mail-tester" and email contents of "mail-test"
    Code (Text):
    echo "mail-test" | mail -s "mail-tester" web-1a0xv@mail-tester.com
    

    If you use Amazon AWS SES smtp Postfix relay on Centmin Mod server, then you also need to send from email address that is verified with Amazon AWS SES i.e. user@yourdomain.com
    Code (Text):
    echo "mail-test" | mail -s "mail-tester" -r user@yourdomain.com web-1a0xv@mail-tester.com
    

    Otherwise, your Postfix /var/log/maillog might have an entry with message like below:
    Code (Text):
    status=bounced (host email-smtp.us-east-1.amazonaws.com[52.206.145.59] said: 554 Message rejected: Email address is not verified. The following identities failed the check in region US-EAST-1
    

    Which means on Amazon AWS SES dashboard, you need to ensure the email and/or domain is verified for user@yourdomain.com or yourdomain.com


    To Test PHP mail() Sent Email

    To test PHP mail() function sent email, create a test-mail.php file (can be any name you want) with contents - changing the web-1a0xv@mail-tester.com to whatever you see on mail-tester.com web site
    PHP:
    <?php
        ini_set
    ('display_errors'1);
        
    error_reporting(-1);
        
    mail ('web-1a0xv@mail-tester.com''Postfix Test''A test email') || print_r(error_get_last());
    ?>
    save test-mail.php to /home/nginx/domains/domain.com/public/ web root directory and then open up domain.com/test-mail.php on your server. Remember to delete test-mail.php afterwards.

    Then click the 'then check your score' button on mail-tester.com site. It will check the email you sent and all it's email headers and give you a report card for the sending domain which is usually your main hostname in default Centmin Mod setups i.e. if sending email is root@hostname.domain.com, then sending domain is hostname.domain.com. You can ignore for subdomains the warning of no MX records for subdomain/hostnames as subdomain/hostnames don't require MX records only the @domain.com domain for receiving emails. This main hostname domain hostname.domain.com would be important for emails sent from your Centmin Mod server so need the required SPF, DKIM, DMARC and PTR DNS records mentioned above.

    This checks php mail() from server sent From = root@host.domain.com where host.domain.com is your main hostname. To test php mail() with a proper From = your@domain.com which maybe set in your PHP web app like Xenforo, Wordpress etc. Use a slight variation of the test-mail.php php file below - changing the web-1a0xv@mail-tester.com to whatever you see on mail-tester.com web site and changing YOURFROM@domain.com email address to the one you set in your PHP web app like Xenforo, Wordpress etc:
    PHP:
    <?php
        ini_set
    ('display_errors'1);
        
    error_reporting(-1);
       
    $headers "From: YOURFROM@domain.com";
        
    mail ('web-1a0xv@mail-tester.com''Postfix Test''A test email',$headers) || print_r(error_get_last());
    ?>
    then open up domain.com/test-mail.php on your server. Then click the 'then check your score' button on mail-tester.com site

    You can also test your @domain.com emails via SSH by adding the From email -r
    Code (Text):
    echo "mail-test" | mail -s "mail-tester" -r user@domain.com your@gmail.com
    

    Then checking your Gmail account including spam box for the email and looking at 'Show Original' for full headers.

    You can also test other target emails i.e. your @gmail.com account via SSH command
    Code (Text):
    echo "mail-test" | mail -s "mail-tester" your@gmail.com
    

    Then checking your Gmail account including spam box for the email and looking at 'Show Original' for full headers.

    Example of Gmail received email's 'Show Original' headers where my centminmod.com domain has working SPF, DKIM and DMARC DNS records setup from my 3rd party Google App/Gsuite email provider service. This is from email sent from my server where from email field was @centminmod.com.

    [​IMG]

    • My centminmod.com SPF record. For main hostname, use the SPF tools and generators linked to here to create your SPF record or use addons/opendkim.sh to generator both DKIM + SPF records. For @domain.com your 3rd party email provider would provide those for you i.e. Google Gsuite/Google App, Zoho, Outlook/Microsoft etc.
    • My centminmod.com DKIM record you need to specify the selector, so if you setup DKIM record as google._domainkey.centminmod.com, then the selector is google and for mxtoolbox.com DKIM check you'd enter it in format of domain:selector i.e. centminmod.com:google. For main hostname, Centmin Mod users can use addons/opendkim.sh to generate both the DKIM + SPF records. For @domain.com your 3rd party email provider would provide those for you i.e. Google Gsuite/Google App, Zoho, Outlook/Microsoft etc.
    • My centminmod.com DMARC record which is just a TXT DNS record so can be looked up as a TXT record as well. You can create DMARC records via free service at https://dmarc.postmarkapp.com/ but DMARC only works when you have both valid working SPF and DKIM DNS records. On the free service site, on the 'Send reports about this domain' field enter your domain.com or subdomain.domain.com (i.e. main hostname host.domain.com) in the field. You can use https://www.learndmarc.com/ to test your DMARC, SPF, DKIM setup is correct for sending email domains.

    Notes For Web App Sent Emails



    Now if you use a web app like Wordpress, Xenforo, IP.Board, Magento, etc to sent emails. Then emails can either be sent as is via Centmin Mod Postfix MTA server via root@host.domain.com where receiving email servers would evaluate the SPF, DKIM, DMARC and PTR DNS records for sending domain @host.domain.com. If server's main hostname i.e. host.domain.com has valid DNS records, then email will be less likely sent to spam box or being rejected by receiving email server.

    If however, your web app allows you to set a from email field and you sent emails from say user@domain.com, then receiving email servers would evaluate the SPF, DKIM, DMARC and PTR DNS records for sending domain @domain.com so your domain.com would need to have those DNS records setup. The above Gmail 'Show Original' email headers screenshot above is an example of such cases.

    Using above mail-tester.com online tool is easiest way to check if all those SPF, DKIM, DMARC and PTR DNS records for sending domain @host.domain.com and @domain.com are setup correctly.

    Main Host Name Setup



    There is a reason the official Centmin Mod Getting Started Guide Step 1 outlines a proper main hostname SPF, DKIM, DMARC and PTR DNS records setup, as it is used for many purposes such as statistics hosting on main hostname vhost, phpmyadmin.sh installer setup of phpmyadmin on main hostname but also for ensuring proper email delivery from main hostname for emails sent from the server.

    As such your main hostname for your Centmin Mod server needs to be unique. The main hostname has a separate Nginx vhost config file at /usr/local/nginx/conf/conf.d/virtual.conf where server_name defined there is auto populated to the server main hostname.

    You can check if the main hostname is uniquely setup via this grep command in SSH type
    Code (Text):
    grep -rw server_name /usr/local/nginx/conf/conf.d/ | grep -w "$(hostname)"
    

    Ensure only 1 entry match is found and that is usually the one contained in main hostname's Nginx vhost config file at /usr/local/nginx/conf/conf.d/virtual.conf

    example output of 1 entry match you should expect as host.domain.com should only be referenced once in server_name for main hostname's Nginx vhost config file at /usr/local/nginx/conf/conf.d/virtual.conf
    Code (Text):
    grep -rw server_name /usr/local/nginx/conf/conf.d/ | grep -w "$(hostname)"
    /usr/local/nginx/conf/conf.d/virtual.conf:            server_name host.domain.com;
    


    If they don't match or if more than one Nginx vhost file is returned from the grep search, then you need to change and update DNS for the main hostname as per Centmin Mod Getting Started Guide Step 1 and then edit the server_name listed in /usr/local/nginx/conf/conf.d/virtual.conf to the newly changed main hostname.

    The main hostname domain i.e. host.domain.com is the domain that server sent emails via Postfix MTA is sent from, so needs to properly setup on server and with proper DNS records.

    Postfix pflogsumm Stats



    Centmin Mod by default installs pflogsumm tool for Postfix log statistics. Centmin Mod only provides the tool for your usage, interpretation and understanding of the output is left up to you to figure out.

    There's a default alias command you can type for today only stats, postfixlog

    The alias for postfixlog command is
    Code (Text):
    alias postfixlog='pflogsumm -d today --verbose_msg_detail /var/log/maillog'
    

    which is basically same as calling pflogsumm directly as
    Code (Text):
    pflogsumm -d today --verbose_msg_detail /var/log/maillog
    

    There's also other commands you can directly use to by invoking pflogsumm command.

    To see yesterday's stats instead of today
    Code (Text):
    pflogsumm -d yesterday --verbose_msg_detail /var/log/maillog
    

    Report for all /var/log/maillog* in sorted time ascending order
    Code (Text):
    pflogsumm $(ls -rt /var/log/maillog*)
    

    To filter on specific date of the month, i.e. Mar 26
    Code (Text):
    grep 'Mar 26' /var/log/maillog | pflogsumm --verbose_msg_detail
    


    Code (Text):
    grep 'Mar 26' /var/log/maillog | pflogsumm --verbose_msg_detail
    
    Grand Totals
    ------------
    messages
    
         23   received
         24   delivered
          0   forwarded
          0   deferred
          0   bounced
          0   rejected (0%)
          0   reject warnings
          0   held
          0   discarded (0%)
    
      24609   bytes received
      27282   bytes delivered
          3   senders
          2   sending hosts/domains
          5   recipients
          3   recipient hosts/domains
    
    
    Per-Hour Traffic Summary
    ------------------------
        time          received  delivered   deferred    bounced     rejected
        --------------------------------------------------------------------
        0000-0100           0          0          0          0          0
        0100-0200           0          0          0          0          0
        0200-0300           0          0          0          0          0
        0300-0400           0          0          0          0          0
        0400-0500           3          3          0          0          0
        0500-0600           2          2          0          0          0
        0600-0700           2          2          0          0          0
        0700-0800           2          2          0          0          0
        0800-0900          10         11          0          0          0
        0900-1000           3          3          0          0          0
        1000-1100           1          1          0          0          0
        1100-1200           0          0          0          0          0
        1200-1300           0          0          0          0          0
        1300-1400           0          0          0          0          0
        1400-1500           0          0          0          0          0
        1500-1600           0          0          0          0          0
        1600-1700           0          0          0          0          0
        1700-1800           0          0          0          0          0
        1800-1900           0          0          0          0          0
        1900-2000           0          0          0          0          0
        2000-2100           0          0          0          0          0
        2100-2200           0          0          0          0          0
        2200-2300           0          0          0          0          0
        2300-2400           0          0          0          0          0
    
    Host/Domain Summary: Message Delivery
    --------------------------------------
     sent cnt  bytes   defers   avg dly max dly host/domain
     -------- -------  -------  ------- ------- -----------
         16    21656        0     0.1 s    0.1 s  host.domain.com
          6     4891        0     0.9 s    1.1 s  domain.com
          2      735        0     2.0 s    2.1 s  mail-tester.com
    
    Host/Domain Summary: Messages Received
    ---------------------------------------
     msg cnt   bytes   host/domain
     -------- -------  -----------
         21    21726   host.domain.com
          2     2883   domain.com
    
    Senders by message count
    ------------------------
         19   root@host.domain.com
          2   user+host@domain.com
          2   nginx@host.domain.com
    
    Recipients by message count
    ---------------------------
         16   root@host.domain.com
          3   user+postfixx@domain.com
          2   user+host@domain.com
          2   web-zxxxc@mail-tester.com
          1   username@domain.com
    
    Senders by message size
    -----------------------
      20991   root@host.domain.com
       2883   user+host@domain.com
        735   nginx@host.domain.com
    
    Recipients by message size
    --------------------------
      21656   root@host.domain.com
       2883   user+host@domain.com
       1512   user+postfixx@domain.com
        735   web-zxxxc@mail-tester.com
        496   username@domain.com
    
    message deferral detail: none
    
    message bounce detail (by relay): none
    
    message reject detail: none
    
    message reject warning detail: none
    
    message hold detail: none
    
    message discard detail: none
    
    smtp delivery failures: none
    
    Warnings: none
    
    Fatal Errors: none
    
    Panics: none
    
    Master daemon messages: none
    
     
    Last edited: Feb 1, 2022
  5. eva2000

    eva2000 Administrator Staff Member

    53,531
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    11:08 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Updated above guide's DMARC mention for using https://www.learndmarc.com/ to test your DMARC, SPF, DKIM setup is correct for sending email domains :D
     
  6. eva2000

    eva2000 Administrator Staff Member

    53,531
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    11:08 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    This thread is even more important now as it outline steps above to ensure email deliverability from your server and domains as Google and Yahoo are stepping up their email deliverability requirements from https://netcorecloud.com/blog/bulk-email-marketing-gmail-yahoo-updates/. Be sure to use Google Postmaster to check your domain's email sending reputation https://community.centminmod.com/th...oesnt-end-up-in-spam-inboxes.6999/#post-37647

     
  7. eva2000

    eva2000 Administrator Staff Member

    53,531
    12,134
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,677
    Local Time:
    11:08 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Just got an email reminding me that Google and Yahoo's new spam enforcement requirements for bulk email sending starts enforcement this month of June 2024. Just as well I developed a tool to clean and verify my email address lists for myself and my paid clients GitHub - centminmod/validate-emails: Self-hosted email verification script to clean up bad invalid email address lists. Supports various commercial email verification provider APIs all in one script :D

    Email sender guidelines FAQ - Google Workspace Admin Help
     
Thread Status:
Not open for further replies.