1. Andrew's Corner :

Using Mutt with Gmail

This page is a guide to using the email client mutt, in conjunction with fetchmail, procmail and msmtp, to send, receive and read email under Linux using a Gmail account as a relay. If you need to ask why anybody would want to put such an effort into an undertaking like this, and did not simply use the web interface of Gmail or even mutt with IMAP, perhaps this page is not for you. To use another's words:

Mutt is not designed to suit everyone, including those without basic knowledge about the Mail Concept, or those unwilling to perform configuration. mutt is even not designed for the mass of "average users", although it works well for them, too. For those who prefer a client to "just work somehow" rather than give dedicated best performance, other mail programs are probably better suited.

Perhaps you already feel the lure of the incredibly powerful Linux console applications? If so read on and please use the email link at the base of this page to let me know how you profited, or otherwise, from the information on this page.

Contents of the Page

This is a fairly complex page that I believe would benefit from reading from top to bottom but I include some navigation here for those who would like to sample rather than consume:

I will not be dealing with the specifics of downloading and installing the required software for this page as there are too many distro-specific issues there. But obviously you will need to have installed OpenSSL, fetchmail, procmail, msmtp and mutt; all fairly standard Linux programs. Whether this is done by using your distro's version or compiling from source the end result should be the same. But before moving on to the actual setup I need to introduce my new friend John.

Introducing John ...

To avoid confusion in editing the many configuration files involved in this setup I will describe the setup of mutt and gmail for my new friend John, who has been created especially for this guide. John's details are as follows:

Gmail Address:        john.example@gmail.com
Gmail Password:       rover
Computer Username:    john 

John's details will always be in italics, bold and in red to remind you, Gentle Reader, to substitute your details for his. Hopefully this will lessen the confusion that I have unintentionally created with older versions of this page!

Downloading the Mail

Downloading the mail is perhaps the most complex part of this exercise but take it one step at a time, Gentle Reader, and it becomes quite logical. First step is to download and setup the required SSL certificates, then to setup fetchmail and finally to setup procmail.

Download the SSL Certificates

The debacle of the Gmail "expired certificate" saga of August 2008 has prompted me to completely rewrite this section and to reconsider the method I have advocated in the past. I will demonstrate a method to create your own certificate pack directly from the Mozilla's source tree using Daniel Stenberg's amazing perl script and then extract the required certificates from this certificate pack. This script is part of the newest version of cURL but I give the directions here to download it manually from gitweb as certainly on Slackware at least this marvellous script is not installed by default. To subsequently run this script you will need perl installed as well as the perl-libwww modules which can be easily picked up by installing Bundle::LWP:

$ mkdir -pv $HOME/.certs
$ cd $HOME/.certs
$ touch Thawte_Premium_Server_CA.pem
$ touch Equifax_Secure_CA.pem
$ wget --no-check-certificate \
       https://github.com/bagder/curl/raw/master/lib/mk-ca-bundle.pl
$ perl mk-ca-bundle.pl

This will generate a full certificate pack named ca-bundle.crt. You will need to open this with your favourite ext editor and then manually extract two certificates, the first is the "Thawte Premium Server CA" certificate which should copied and pasted into the Thawte pem file that we created with touch. I show the certificate here for any who has trouble with the perl script, although it is always best to generate the certificate yourself:

-----BEGIN CERTIFICATE-----
MIIDJzCCApCgAwIBAgIBATANBgkqhkiG9w0BAQQFADCBzjELMAkGA1UEBhMCWkExFTATBgNVBAgT
DFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYDVQQKExRUaGF3dGUgQ29uc3Vs
dGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UE
AxMYVGhhd3RlIFByZW1pdW0gU2VydmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNlcnZl
ckB0aGF3dGUuY29tMB4XDTk2MDgwMTAwMDAwMFoXDTIwMTIzMTIzNTk1OVowgc4xCzAJBgNVBAYT
AlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEdMBsGA1UEChMU
VGhhd3RlIENvbnN1bHRpbmcgY2MxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2
aXNpb24xITAfBgNVBAMTGFRoYXd0ZSBQcmVtaXVtIFNlcnZlciBDQTEoMCYGCSqGSIb3DQEJARYZ
cHJlbWl1bS1zZXJ2ZXJAdGhhd3RlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA0jY2
aovXwlue2oFBYo847kkEVdbQ7xwblRZH7xhINTpS9CtqBo87L+pW46+GjZ4X9560ZXUCTe/LCaIh
Udib0GfQug2SBhRz1JPLlyoAnFxODLz6FVL88kRu2hFKbgifLy3j+ao6hnO2RlNYyIkFvYMRuHM/
qgeN9EJN50CdHDcCAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQQFAAOBgQAm
SCwWwlj66BZ0DKqqX1Q/8tfJeGBeXm43YyJ3Nn6yF8Q0ufUIhfzJATj/Tb7yFkJD57taRvvBxhEf
8UqwKEbJw8RCfbz6q1lu1bdRiBHjpIUZa4JMpAwSremkrj/xw0llmozFyD4lt5SZu5IycQfwhl7t
UCemDaYj+bvLpgcUQg==
-----END CERTIFICATE-----

The second is the "Equifax Secure CA" certificate which should copied and pasted into the Equifax pem file that we created with touch. Again I show the certificate here for any who has struggled with the perl script:

-----BEGIN CERTIFICATE-----
MIIDIDCCAomgAwIBAgIENd70zzANBgkqhkiG9w0BAQUFADBOMQswCQYDVQQGEwJVUzEQMA4GA1UE
ChMHRXF1aWZheDEtMCsGA1UECxMkRXF1aWZheCBTZWN1cmUgQ2VydGlmaWNhdGUgQXV0aG9yaXR5
MB4XDTk4MDgyMjE2NDE1MVoXDTE4MDgyMjE2NDE1MVowTjELMAkGA1UEBhMCVVMxEDAOBgNVBAoT
B0VxdWlmYXgxLTArBgNVBAsTJEVxdWlmYXggU2VjdXJlIENlcnRpZmljYXRlIEF1dGhvcml0eTCB
nzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAwV2xWGcIYu6gmi0fCG2RFGiYCh7+2gRvE4RiIcPR
fM6fBeC4AfBONOziipUEZKzxa1NfBbPLZ4C/QgKO/t0BCezhABRP/PvwDN1Dulsr4R+AcJkVV5MW
8Q+XarfCaCMczE1ZMKxRHjuvK9buY0V7xdlfUNLjUA86iOe/FP3gx7kCAwEAAaOCAQkwggEFMHAG
A1UdHwRpMGcwZaBjoGGkXzBdMQswCQYDVQQGEwJVUzEQMA4GA1UEChMHRXF1aWZheDEtMCsGA1UE
CxMkRXF1aWZheCBTZWN1cmUgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MQ0wCwYDVQQDEwRDUkwxMBoG
A1UdEAQTMBGBDzIwMTgwODIyMTY0MTUxWjALBgNVHQ8EBAMCAQYwHwYDVR0jBBgwFoAUSOZo+SvS
spXXR9gjIBBPM5iQn9QwHQYDVR0OBBYEFEjmaPkr0rKV10fYIyAQTzOYkJ/UMAwGA1UdEwQFMAMB
Af8wGgYJKoZIhvZ9B0EABA0wCxsFVjMuMGMDAgbAMA0GCSqGSIb3DQEBBQUAA4GBAFjOKer89961
zgK5F7WF0bnj4JXMJTENAKaSbn+2kmOeUJXRmm/kEd5jhW6Y7qj/WsjTVbJmcVfewCHrPSqnI0kB
BIZCe/zuf6IWUrVnZ9NA2zsmWLIodz2uFHdh1voqZiegDfqnc1zqcPGUIWVEX/r87yloqaKHee95
70+sB3c4
-----END CERTIFICATE-----

And finally to rehash the certificates so Openssl knows how to find them:

$ c_rehash $HOME/.certs/

I should mention as well that many distros have a bundle of CA certificates with a package name something like "ca-certificates" that can also be used in place of these reasonably complex instructions. I shall point out how this bundle can be utilised further on in the guide. This completes the SSL configuration and now onto lighter matters with the required steps to setup fetchmail.

Setting up fetchmail

Fetchmail is fairly easy to use and setup, particularly as the hard yards have already been done with the SSL certificates. You will need to create the file $HOME/.fetchmailrc and then add the following information for accessing the Gmail server:

poll pop.gmail.com                   
with proto POP3                      
user 'john.example'        
there with password 'rover'        
is 'john' here                              
mda "/usr/bin/procmail -d %T"        
options                                                             
no keep                                 
ssl                                  
sslcertck                            
sslcertpath /home/john/.certs/

It is almost too obvious to point out, Gentle Reader, that you will need to substitute your own details for password, username, email address etc. Those who have installed a CA certificate bundle through their distro's repository will need to change the sslcertpath setting to somehing like /etc/ssl/certs according to the installation path of the certificate bundle. There remains a final touch, since the username and password are openly in this file you should make the file readable only by the file owner:

$ chmod 600 ~/.fetchmailrc

Now would be a good to time also to make sure you have POP forwarding enabled in your Gmail account. You will find this in: Settings - Forwarding and POP at the web interface of Gmail. Note as well that you cannot remove mail from Gmail servers via POP3 but you can choose to have your messages archived, kept or deleted once they have been downloaded via POP3. This is a Gmail setting hidden in Settings - Forwarding and POP: "When messages are accessed with POP..."

Setting up procmail

procmail is the final link in the chain for downloading mail. Before setting it up there is a little bit of outstanding business to attend to: setting the $MAIL environmental variable so that all software that deals with mail knows where the delivery point is. Add the following to ~/.bashrc making the obvious substitution for username:

# Sets the Mail Environment Variable
MAIL=/var/spool/mail/john && export MAIL

procmail will now know the location of the default mail spool and will deliver all mail to there that has not been sorted to other locations. But the creation of a $HOME/.procmailrc is still required and I give an example below. I have also added a sample filtering recipe for the mutt user mailing list to give the very beginning of what can be a complex process:

PATH=/bin:/usr/bin:/usr/local/bin 
VERBOSE=off  
DEFAULT=/var/spool/mail/john            
MAILDIR=$HOME/mail            
LOGFILE=$HOME/.procmaillog  
# Recipes below this comment:

:0:
* ^TOmutt-user
mutt

The single filtering recipe means that procmail delivers all email addressed to "mutt-user" directly to $HOME/mail/mutt. All other mail goes to the default location /var/spool/mail/username as specified by $MAIL environmental variable. If for some reason you are averse to adding this to ~/.bashrc, Gentle Reader, I have demonstrated above how to add this to ~/.procmailrc with the DEFAULT setting. For the definitive guide to further recipes, and guidance on much more complex recipes, don't forget to run man procmailex, it all starts to make sense after a while.

Sending the Mail

I have formerly been an advocate for the simple MTA ssmtp, which some would call a Mail Sending Agent (MSA). However I believe that development of ssmtp has ceased and I have moved with some regret on to msmtp. A single configuration file is required for msmtp: $HOME/.msmtprc and the following section gives the required details to access Gmail and reference the required certificate:

account default              
host smtp.gmail.com          
port 587                     
from john.example@gmail.com     
tls on                       
tls_starttls on              
# tls_trust_file /home/john/.certs/Thawte_Premium_Server_CA.pem
tls_trust_file /home/john/.certs/Equifax_Secure_CA.pem
auth on                     
user john.example        
password rover          
logfile ~/.msmtp.log

I need not mention again, Gentle Reader, that there should be some fairly obvious changes here to substitute your own username, password and email address? For those who have installed a bundle of CA certificates from their distro's repository the tls_trust_file setting will need to be changed to something like /etc/ssl/certs/ca-certificates.crt depending on the exact installation path of the certificate bundle. And then the final touch, since the username and password are openly in this file, you should make the file readable only by the file owner:

$ chmod 600 ~/.msmtprc

msmtp is a great program that has many features that quite frankly I am still exploring, feel free to point out anything that I have missed, there is an email link at the base of this page for that purpose. But now finally to Mutt:

Reading the Mail

September 15th 2010 saw the release of a "Development" version of Mutt: 1.5.21 and it is this version that I have used for this page. Although labelled as a pre-release version I have found it very stable and I suspect you will as well. Mutt is driven by the file $HOME/.muttrc and I spent some time building this file from scratch but for you, Gentle Reader, I include here a more basic version, similar to the one I started from. Some parts of this, such as aliases and colors, are sourced from their own file so don't forget to create these files.

#======================================================#
# Boring details
set realname = "john"
set from = "john.example@gmail.com"
set use_from = yes
set envelope_from ="yes"

# Use a signature
set signature="~/.signature"

# Use msmtp rather than sendmail. Check that 
# the path is correct for your system:
set sendmail="/usr/bin/msmtp"   

# If not set in ~/.bashrc:
set spoolfile = /var/spool/mail/john

#======================================================#
# Folders
set folder="$HOME/mail"      # Local mailboxes stored here
set record="+sent"           # Where to store sent messages
set postponed="+postponed"   # Where to store draft messages
set mbox_type=mbox           # Mailbox type
set move=no                  # Don't move mail from spool

#======================================================#
# Watch these mailboxes for new mail, useful only if 
# Procmail or Maildrop is used to sort mail.
mailboxes ! +slrn +fetchmail +mutt
set sort_browser=alpha    # Sort mailboxes by alpha(bet)

#======================================================#
# What to show and order of headers
ignore *
unignore Date: From: User-Agent: X-Mailer X-Operating-System To: \
         Cc: Reply-To: Subject: Mail-Followup-To:
hdr_order Date: From: User-Agent: X-Mailer X-Operating-System To: \
        Cc: Reply-To: Subject: Mail-Followup-To:
                       
#======================================================#
# which editor do you want to use? 
# vim of course!
set editor="vim -c 'set tw=70 et' '+/^$' " 
set edit_headers=yes      # See the headers when editing

#======================================================#
# Aliases

set alias_file = ~/mail/mutt_aliases # In their own file
source ~/mail/mutt_aliases           # Source them
set sort_alias=alias                 # Sort alphabetically

#======================================================#
# Colours: defaults are a little bleak so experiment!

source ~/mutt/mutt_colors            # In their own file 

#======================================================#
# Lists: An example using the mutt-users list:

lists mutt-users
subscribe mutt-users
set followup_to=yes        # Sets 'Mail-Followup-To' header
set honor_followup_to=yes  
fcc-hook mutt-user +mutt   # See your own posts using fcc

#======================================================#
# Odds and ends

set markers          # mark wrapped lines of text in the pager with a +
set smart_wrap       # Don't wrap mid-word
set pager_context=5  # Retain 5 lines of previous page when scrolling.
set status_on_top    # Status bar on top.
push <show-version>  # Shows mutt version at startup

I have been taken to task somewhat by one reader of this page who felt there should be a little more information about Mutt and colors and so ...

Coloring in Mutt

If your terminal supports color, and I believe it is a rarity these days for this not to be case, you can color almost any aspect of the Mutt window. All the fine details are in the Mutt manual: "Section 3: Configuration 8: Using color and mono video attributes". But I can tell you, Gentle Reader, that the basic usage is:

color   object   foreground   background

This can be made a lot more compilcated but a simple start is a good start. The basic colors are white, black, green, magenta, blue, cyan, yellow, red and default but you can also prefix a foreground color with "bright" to make the color bold. Now I personally use a Terminal with a white background so my own colors are:

#---- Mutt Colors for White Background -------
color    hdrdefault    black           default   
color    quoted        red             default   
color    signature     brightblack     default   
color    indicator     brightwhite     red
color    attachment    black           default
color    error         red             default   
color    message       blue            default   
color    search        brightwhite     magenta
color    status        brightyellow    blue
color    tree          red             default   
color    normal        blue            default   
color    tilde         green           default   
color    bold          brightyellow    default   
color    markers       red             default

There should also be a file called colors.default installed as part of your Mutt installation which is intended for Terminals with white backgrounds, this might also furnish a starting point as it did once for me. Just to balance out the equation a little I will also give the colors for those who prefer a black background. This example is taken directly from another sample file colors.linux which should be installed along with Mutt:

#---- Mutt Colors for Black Background -------
color   hdrdefault   blue              black
color   quoted       blue              black
color   signature    blue              black
color   attachment   red               black
color   message      brightred         black
color   error        brightred         black
color   indicator    black             red
color   status       brightgreen       blue
color   tree         white             black
color   normal       white             black
color   markers      red               black
color   search       white             black
color   tilde        brightmagenta     black
color   index        blue              black ~F
color   index        red               black "~N|~O"

If you don't like either of these feel free to experiment a little and come up with your own, most Mutt users will alter these basic colors. I prefer mine simple but if you pull out the manual you will see that you can spend many hours getting it exactly as you want. But now finally to check the mail:

Reward Time!

Finally it is reward time as you open Mutt, type ! to open a shell prompt, type fetchmail -v and start reading the mail! My parting gift to you, Gentle Reader, is a little macro that was written for me by a generous person on the mutt-user mailing list that will actually do this for you when you simply press "I". Place the following in your ~/.muttrc file:

macro index,pager I '<shell-escape> fetchmail -v<enter>'

This ~/.muttrc is intentionally a little basic, although I suspect that it will cover most needs anyway. Don't be afraid however to spend hour after hour painstakingly crafting your own to produce that perfect setup as mutt encourages and rewards such efforts!

And in conclusion...

I wish you all the best with one of the truly great Linux console programs! Please send me an email, using Mutt of course, to let me know if you have found this page at all useful and as well to suggest any corrections that you feel should be made. If you are feeling generous perhaps you could also assist me keep this page alive by assisting with the hosting bills for this site, if not please feel free to utilise this page in any way you see fit and remember: "Have Fun!".