How Do I Secure My Mail Server? A Comprehensive Guide

Securing Inbound Email Traffic

Encrypting a mail server and encrypting email traffic are actually two different things. A secure email server requires encryption during transfer, encryption of email, and encryption of saved emails. 

End User Side Encryption

PGP/MIME and S/MIME are two options for encrypting emails end-to-end. These two options use certificate-based encryption for emails from the moment they are originating from the end user device until they are received on the recipient’s end user device.. 

S/MIME uses a public key or asymmetric cryptography as well as digital certificates for emails. Certificates help authenticate the email sender. 

Authentication Credentials Encryption

Axigen, as one of the leading email server software providers, uses CRAM-MD5, DIGEST-MD5, and GSSAPI for email credentials encryption. Read more about Axigen mail server security on our dedicated page.

SMTP Submission Authentication is required to properly identify the sender and to ensure that your email server does not become an open relay abused by 3rd parties.

For email in-transit encryption, TLS is the de facto standard. It can and should be used to secure traffic for webmail, IMAP, and any other client access protocols. 

SMTP Services

Simple Mail Transfer Protocol (or SMTP) is the protocol of choice used by most email clients to submit messages to an email server as well as emails servers sending / relaying messages from one server to another on their way to their designated user.

Here are the most commonly occurring security issues when transmitting emails:

• Unauthorized access to your emails and data leakage
• Spam and Phishing
• Malware
• DoS attacks

SSL (Secure Sockets Layer) is a cryptographic protocol developed by Netscape in 1995 designed to provide enhanced security over network communications and it is the predecessor of TLS (Transport Layer Security). Since all SSL versions currently have a lot of known and exploitable vulnerabilities is no longer recommended for production use. Securing transmission with TLS is the current de facto standard: recommended TLS versions are 1.1, 1.2 and, the latest and most secure, 1.3.

SSL/TLS encrypts the messages between the email client and the email server as well as between email servers. If  the encrypted SMTP communication is recorded by a malicious third party, that party will only see what seems to be random characters that replace the email content which means your contacts and message data is still protected and unreadable. 

Axigen also supports a TLS extension called Perfect Forward Secrecy which is a feature of specific key agreement protocols that gives assurances that session keys will not be compromised even if long-term secrets used in the session key exchange are compromised. In layman terms, should the private keys stored on the server be lost or breached, previously recorded encrypted SMTP sessions are still indecipherable.

DNSBL and URIBL

Domain Name System Blacklist (DNSBL) or Real-time Blackhole List (RBL) is in essence a service that provides a black list of known domains and IP addresses that have a reputation of being a source of spam. Typically mail server software can be configured to check one or more of these listings.

A DNSBL is more of a software mechanism, rather than a specific list. There are many in existence, which use a wide array of criteria that might get an address listed or unlisted: listing the addresses of machines being used to send spam, internet service providers (ISPs) are known to host spammers, etc.

• The Spamhaus DBL is a service that blacklists domains found in spam messages and listed as having a poor reputation.
• The URIBL service is a list of domains detected as sending spam email

DNSBL servers are blacklisted as spammers, and when you define a server as one, emails from such servers are automatically dropped.

Axigen also offers two such services to their customers: aDNSBL and aURIBL, these two are premium IP based DNSBL and URIBL lists, operated and curated by Axigen and they can be used by Axigen customers that subscribe to these optional services.

SPF, DKIM, and DMARC

SPF

SPF (Sender Policy Framework) is a DNS TXT entry that has a list of servers that should be considered as having been allowed to send mail on behalf of a specific domain. Being a DNS entry can be considered as a way to enforce the fact that the entry list is trustworthy for the domain as the only people allowed to add or change that domain zone are the owners or administrators of the domain. 

More information about configuring SPF for the Axigen services can be accessed here.

DKIM

DKIM (DomainKeys Identified Mail) is a method to verify that the messages’ content is trustworthy, showing that the content wasn’t modified from the moment that the message left the initial mail server and until it reached the destination. This additional layer of consistency is accomplished by the use of a standard public / private key signing process. As in the case of the SPF the owners or administrators of the domain add a DNS record that contains the public DKIM key that will be used by receivers to verify that the message DKIM signature is correct, and on the sender side of things the server will use the private key corresponding to the public key present in the DNS record to sign the mail messages. 

More information about configuring DKIM for Axigen services can be accessed here.

DMARC

DMARC (Domain-based Message Authentication, Reporting, and Conformance) is a protocol that uses SPF and DKIM to determine if the email message is authentic. In essence it makes it easier for ISPs to prevent malicious third parties from performing practices such as domain spoofing to phish for users private information.

DMARC states a clear policy about both SPF and DKIM and allows the setting of an address which should be used to send reports about the mail messages sent by the server. This policy should be used by all receiving servers and clients.

More information about configuring DMARC for Axigen services can be accessed here.

How SPF, DKIM, DMARC Work

All these tools rely heavily on DNS and the way they work after all the set up has been taken care of is as follows:

SPF

• on receipt, the HELO message and the address of the sender are retrieved by the mail server
• the mail server fetches a TXT DNS query against the messages’ domain SPF entry
• the retrieved SPF entry data is then used to verify the sending server
• if this check fails the message will be rejected with information about the rejection

DKIM

• On sending a message, the last server in the domain infrastructure checks against its internal settings if the domain that is used in the “From:” header is indeed included in its “signing table”. If this check fails the everything stops here
• A header named “DKIM-Signature”  is added to the message header list by generating a signature using the private part of the key on the content of the message
• After this point the message's main content cannot be modified or the DKIM-Signature header won’t be correct anymore and authentication will fail.
• On reception of the message, the receiving server will make a DNS query to retrieve the public key used in the DKIM-Signature
• After that the  DKIM header can be used to decide if the message was changed in transit or if it is trustworthy

DMARC

• upon reception, the receiving server will check if a DMARC policy is published in the domain used by the SPF and / or DKIM checks
• if one or both SPF /  DKIM checks succeed but are not aligned with the  DMARC policy, then the check is considered unsuccessful, otherwise, if they are also aligned with the DMARC policy then the check is successful
• on failure, based on what action is published by the DMARC policy, different actions can be taken

Even if you have a perfectly functional system and all of the above mentioned tools set up and running smoothly, you can’t be 100% safe because not all the servers out there are using these tools. 

Content Filtering

Content filters allow you to scan and inspect incoming / outgoing messages and take corresponding actions based on the results automatically.

Services such as these mainly scan the content of the email message and decide whether the content matches spam filters and blocks the message from reaching the inbox. Scans also look at image metadata and headers as well as the message text content.

Securing Outbound Email Traffic

Send and Receive Restrictions

Limits can be applied to the messages that are sent by the users you host on your email server. You can control the maximum size that a message can have in its entirety or the size of a message’s individual parts or even both of these things. For example, you can control the maximum size of the message header or it’s attachments, or set a limit for the  maximum number of recipients that a user can add to an outgoing message. 

Furthermore, and more importantly, as an administrator, you can create sending quotas (with exceptions) that ensure that your Fair Usage Policy is enforced automatically. 

You can read more about configuring these restrictions in Axigen WebAdmin here.

Outbound Spam Protection

Having control over what goes out of your email servers is as important as knowing what comes in. So having a policy to scan the outgoing messages as well as the incoming messages is important because it can stop someone from sending spam messages and as such attracting unwanted repercussions on you.

More about this subject in my article on LinkedIn here.

Securing Mailbox Access

Webmail Two Factor Authentication (2FA)

Making sure your user accounts are secure even though you are probably using SSL/TLS, is important because sometimes user passwords are not the strongest.

Beside the fact that Axigen support configurable Password Policies, enabling two-factor authentication can greatly improve the account security of each user and protect their data from malicious third parties that might otherwise get access to their account because they may have gotten their password from another service they were using that had a security backdoor.

Axigen provides Two Factor Authentication support for user accounts.

SSL/TLS Listeners

It is very important that your listeners are configured correctly with good SSL versions and cypher suites. The Axigen server comes with everything set up and we recommend you always keep the server up to date to ensure that your SSL listeners are A-grade.

IMAP Encryption and Authentication Recommended Settings

Using an encrypted connection with StartTLS enabled is the best way to ensure that your and your clients data is protected and can’t be read by a malicious third party.

Axigen WebAdmin allows control over the settings of the encryption and authentication of the mail server, you can see the recommended settings for configuring IMAP on this documentation page.

Protecting From Brute Force Attacks

A brute-force attack is a type of cyber-attack where a malicious third party tries different passwords and passphrases using an automated script until they find the right combination to gain access to an account or service. It may have been around for a long time, however it is still very popular because of how effective it is against weak passwords, which is why Two Factor Authentication is an important feature to have on user accounts. 

Fail2Ban (Linux) and RDPGuard (Windows) are intrusion prevention systems that add protection for brute-force attacks to mail servers. By monitoring log files and blocking IP addresses of hosts that perform too many login attempts, or too many connections in a short period of time that is defined by the administrator of the mail server.

More about how to set up your Axigen server to use Fail2Ban on Linux or how to set up your Axigen server to use RDPGuard on Windows

Firewall

One of the critical and truly mandatory network-level security controls is the firewall. A  Firewall should have advanced persistent threat analysis features, as they are capable of detecting zero-day security attacks. It is a best practice to run intrusion detection systems (IDS) as well. An email security gateway is required to screen inbound / outbound email traffic.

Firewall filtering rules can be used to deny / allow specific email traffic. This is useful to stop the server from becoming a relay and sending mass spam emails. Packet filtering rules help stop DDoS and DoS attacks.

Axigen has an internal component for the application-level firewall that handles this for you as part of the server’s security layers.

You can read more about the configuration options available in the WebAdmin for the built-in firewall in the Flow control section of this documentation page.

Conclusion

A secure email server essentially has both network and server level security controls. It is a standard practice to configure and maintain your own email server. However, some organizations choose to buy off-the-shelf email server software solutions. If you consider this option, security should be your highest consideration.

There is no completely secure system anywhere in the world. However, some email software solutions provide comprehensive packages covering security at all layers, including network and server levels.

 A highly secure email server solution should have:

• firewall rules
• secure email gateway
• server-level controls including encryption, anti-spam / anti-phishing / antivirus, as well as a monitoring, analysis service.

One of the top solutions is the secure email server solution offered by Axigen, which you can find out more about here.