IMAP protocol in RFC3501 introduced \Answered flag. \Answered flag should mark messages that have been answered. This flag can by set by client applications or even by SMTP server, which can be examining In-Reply-To email headers.

Unfortunately if the message is answered through Gmail’s web interface, the \Answered flag will not be set. It will only be set if the messages were answered using an email program that sets this flag.

This means that \Answered flag can’t be used to find not-answered emails.

One option to resolve this problem is to use Gmail IMAP extensions, X-GM-THRID in particular.

We’ll connect to Gmail using IMAP – remember to use SSL and be sure to enable IMAP protocol for Gmail.

As we plan to search all emails, including sent and received ones, we’ll use CommonFolders class to get ‘All Mail’ folder.

Then we’ll get Envelope for each message. Envelope contains GmailThreadId property. It contains thread id to which this email was assigned to by Gmail.

Finally we’ll find threads that contain only one message – those are messages that were not answered.

// C#
 
using(Imap client = new Imap())
{
    client.ConnectSSL("imap.gmail.com");
    client.Login("pat@gmail.com", "app-password");
 
    // Select 'All Mail' folder.
    List<FolderInfo> folders = client.GetFolders();
    CommonFolders common = new CommonFolders(folders);
    client.Select(common.AllMail);
 
    // Get envelopes for all emails.
    List<long> uids = client.GetAll();
    List<Envelope> envelopes = client.GetEnvelopeByUID(uids);
 
    // Group messages by thread id.
    var threads = new Dictionary<decimal, List<Envelope>>();
    foreach (Envelope envelope in envelopes)
    {
        decimal threadId = (decimal) envelope.GmailThreadId;
        if (threads.ContainsKey(threadId) == false)
            threads[threadId] = new List<Envelope>();
         
        threads[threadId].Add(envelope);
    }
 
    // Find threads containing single message.
    foreach (KeyValuePair<decimal, List<Envelope>> pair in threads)
    {
        if (pair.Value.Count == 1)
        {
            Envelope envelope = pair.Value[0];
            Console.WriteLine("Gmail message id: {0}; subject: {1}", 
                envelope.GmailMessageId , 
                envelope.Subject);
 
        }
    }
    client.Close();
}

' VB.NET
 
Using client As New Imap()
    client.ConnectSSL("imap.gmail.com")
    client.Login("pat@gmail.com", "app-password")
 
    ' Select 'All Mail' folder.
    Dim folders As List(Of FolderInfo) = client.GetFolders()
    Dim common As New CommonFolders(folders)
    client.[Select](common.AllMail)
 
    ' Get envelopes for all emails.
    Dim uids As List(Of Long) = client.GetAll()
    Dim envelopes As List(Of Envelope) = client.GetEnvelopeByUID(uids)
 
    ' Group messages by thread id.
    Dim threads = New Dictionary(Of Decimal, List(Of Envelope))()
    For Each envelope As Envelope In envelopes
        Dim threadId As Decimal = CDec(envelope.GmailThreadId)
        If threads.ContainsKey(threadId) = False Then
            threads(threadId) = New List(Of Envelope)()
        End If
 
        threads(threadId).Add(envelope)
    Next
 
    ' Find threads containing single message.
    For Each pair As KeyValuePair(Of Decimal, List(Of Envelope)) In threads
        If pair.Value.Count = 1 Then
            Dim envelope As Envelope = pair.Value(0)
 
            Console.WriteLine("Gmail message id: {0}; subject: {1}", _
                envelope.GmailMessageId, envelope.Subject)
        End If
    Next
    client.Close()
End Using

Files containing the .msg file extension are most commonly created by or saved from within one of the Microsoft Outlook email applications.

The MSG file contains information about a saved email file including the date of the message, the subject, who sent it, who received it and the contents of the email associated with the file. If attachments ares included with an email, that information will also be saved within the associated MSG file.

MsgConverter , than can be used to convert .msg files to MIME, is written in pure .NET, it does not require registration or any other components or libraries (including Outlook).

The following code snippet reads .msg file in .NET and saves it using MIME format to disk. MIME is shorthand for Multipurpose Internet Mail Extensions and is an Internet standard used to store and transport email messages.

// C#

using (MsgConverter converter = new MsgConverter(@"c:\outlook1.msg"))
{
    if (converter.Type == MsgType.Note)
    {
        IMail email = converter.CreateMessage();

        // Render message using MIME format to byte array
        byte[] mime = email.Render();

        // Save message to file using MIME message format
        email.Save(@"c:\mime.eml");
    }
}

' VB.NET

Using converter As New MsgConverter("c:\outlook1.msg")
If converter.Type = MsgType.Note Then
    Dim email As IMail = converter.CreateMessage()

	' Render message using MIME format to byte array
	Dim mime As Byte() = email.Render()

	' Save message to file using MIME message format
	email.Save("c:\mime.eml")
    End If
End Using

Files containing the .msg file extension are most commonly created by or saved from within one of the Microsoft Outlook email applications.

The MSG file contains information about a saved email file including the date of the message, the subject, who sent it, who received it and the contents of the email associated with the file. If attachments ares included with an email, that information will also be saved within the associated MSG file.

MsgConverter , than can be used to read .msg files, is written in pure .NET, it does not require registration or any other components or libraries (including Outlook).

The following code snippet reads .msg file in .NET and access its most common properties, such as subject, sender and attachments.

// C#

using (MsgConverter converter = new MsgConverter(@"c:\outlook1.msg"))
{
    if (converter.Type == MsgType.Note)
    {
        IMail email = converter.CreateMessage();

        Console.WriteLine("Subject: {0}", email.Subject);
        Console.WriteLine("Sender name: {0}", email.Sender.Name);
        Console.WriteLine("Sender address: {0}", email.Sender.Address);

        Console.WriteLine("Attachments: {0}", email.Attachments.Count);
        foreach (MimeData attachment in email.Attachments)
        {
            attachment.Save(@"c:\" + attachment.SafeFileName);
        }
    }
}

' VB.NET

Using converter As New MsgConverter("c:\outlook1.msg")
    If converter.Type = MsgType.Note Then
        Dim email As IMail = converter.CreateMessage()

        Console.WriteLine("Subject: {0}", email.Subject)
        Console.WriteLine("Sender name: {0}", email.Sender.Name)
        Console.WriteLine("Sender address: {0}", email.Sender.Address)
        
        Console.WriteLine("Attachments: {0}", email.Attachments.Count)
        For Each attachment As MimeData In email.Attachments
            attachment.Save("c:\" + attachment.SafeFileName)
        Next

    End If
End Using

You can read more on how to access To, Cc, Bcc fields.

There’s often quite a confusion about the different terms: SSL, TLS, STARTTLS and STLS.

SSL and TLS

SSL and TLS are cryptographic protocols, both provide a way to encrypt communication channel between two machines over the Internet (e.g. client computer and a server). SSL stands for Secure Sockets Layer and current version is 3.0. TLS stands for Transport Layer Security and the current version is 1.2. TLS is the successor to SSL. The terms SSL and TLS can be used interchangeably, unless you’re referring to a specific protocol version.

Version numbering is inconsistent between SSL and TLSs. When TLS took over SSL as the preferred protocol name, it began with a new version number. The ordering of protocols in terms of oldest to newest is: SSLv2, SSLv3, TLSv1.0, TLSv1.1, TLSv1.2.

STARTTLS and STLS

STARTTLS is a protocol command, that is issued by an email client. It indicates, that the client wants to upgrade existing, insecure connection to a secure connection using SSL/TLS cryptographic protocol. STARTTLS command name is used by SMTP and IMAP protocols, whereas POP3 protocol uses STLS as the command name.

Clear text/Plain text

No security protocol is used at all. All commands, responses and data are transferred in plain text.

client.Connect("mail.example.com");

Implict SSL mode

Implict SSL mode means, that you connect to SSL/TLS encrypted port.

client.ConnectSSL("mail.example.com");

Explicit SSL mode

Explicit SSL mode means, that you connect to plaint text port and secure the connection by issuing STARTTLS (or STLS) command afterwards (you explicitly secure the connection).

client.Connect("mail.example.com");
client.StartTLS();

Securing the connection

Regardless of whether you use implict (connecting to an SSL/TLS encrypted port) or explicit (using STARTTLS to upgrade an existing connection) mode, both sides will negotiate which protocol and which version to use. This negotiation is based on how client and server have been configured and what each side supports.

SSL/TLS support

Support for SSL/TLS is virtually universal, however which versions are supported is variable. Pretty much everything supports SSLv3. Most machines support TLSv1.0.

TLS vs STARTTLS naming problem

One significant complicating factor is that some email software incorrectly uses the term TLS when they should have used “STARTTLS” or “explicit SSL/TLS”. Older versions of Thunderbird used “TLS” to mean “enforce use of STARTTLS to upgrade the connection, and fail if STARTTLS is not supported” and “TLS, if available” to mean “use STARTTLS to upgrade the connection, if the server advertises support for it, otherwise just use an insecure connection” (very problematic, as we’ll see below).

Port numbers

To add security to some existing protocols (IMAP, POP3, SMTP), it was decided to just add SSL/TLS encryption as a layer underneath the existing protocol. However to distinguish that software should talk the SSL/TLS encrypted version of the protocol rather than the plaintext one, a different port number was used for each protocol:

Too many ports? Solution: Plain text + STARTTLS

At some point, it was decided that having 2 ports for every protocol was wasteful.

Instead it’s better to have 1 port, that starts off as plain text, but clients can upgrade the connection to an SSL/TLS encrypted one, using STARTTLS (or STLS for POP3 protocol) command.

STARTTLS problems

There were a few problems with STARTTLS introduction. There exists lots of software, that used the alternate port numbers with pure SSL/TLS connections. Client software can be very long lived, so you can’t just disable the encrypted ports until all software has been upgraded.

Each protocol received mechanisms to tell clients that the server supported upgrading to SSL/TLS (e.g. STARTTLS in IMAP’s CAPABILITY response), and that they should not attempt to login without doing the STARTTLS upgrade (LOGINDISABLED in IMAP’s CAPABILITY response).

This created two unfortunate situations:

• Some software just ignored the “login disabled until upgraded” announcement (LOGINDISABLED, STARTTLS) and just tried to log in anyway, sending the user login name and password over clear text channel. The server rejected the login and password, but the details had already been sent over the Internet in plain text.
• Other software saw the “login disabled until upgraded” announcement, but then wouldn’t upgrade the connection automatically, and thus reported login errors back to the user, which caused confusion about what was wrong.

Both of these problems resulted in significant compatibility issues with existing clients, and so most system administrators continued to just use plain text connections on one port, and encrypted connections on a separate port number.

Disable plain text for IMAP and POP3

Many companies (e.g. Gmail, Outlook.com) disabled plain IMAP (port 143) and plain POP3 (port 110), so people must use a SSL/TLS encrypted connection – this removes the need for having STARTTLS command completely.

SMTP STARTTLS stays

The one real exception to the above is SMTP. Most email software used SMTP on port 25 to submit messages to the email server for onward transmission to the destination. However SMTP was originally designed for transfer, not submission. So yet another port (587) was defined for message submission.

Port 587 doesn’t mandate requiring STARTTLS, however the use of port 587 became popular around the same time as the realization that SSL/TLS encryption of communications between clients and servers was an important issue. The result is that most systems, that offer message submission over port 587 require clients to use STARTLS to upgrade the connection. Login and password to authenticate is also required.

There has been an additional benefit to this approach as well. By moving users away from using port 25 for email submission, ISPs can block outgoing port 25 connections from users’ computers, which were a significant source of spam, due to user computers infected with spam sending viruses.

Further readings

• Using SSL/TLS with IMAP
• Using SSL/TLS with SMTP
• Using SSL/TLS with POP3

Summary

Outlook.com IMAP implementation does not work correctly with messages containing national characters. ENVELOPE retrieved for a message, that has national characters in the subject (correctly encoded using Base64), has ‘?’ characters instead of encoded characters.

Actual:
ENVELOPE ("Fri, 27 Sep 2013 17:59:48 +0200" "za????"

Expected:
ENVELOPE ("Fri, 27 Sep 2013 18:23:23 +0200" "=?utf-8?B?emHFvMOzxYLEhw==?=" ...
-or-
ENVELOPE ("Fri, 27 Sep 2013 18:21:17 +0200" "=?ISO-8859-2?B?emG/87Pm?=" ...

Logs

Mail.dll C: 6d9d6c00b9294efb APPEND Inbox (\SEEN) {243}
Mail.dll S: + Ready
Mail.dll C: Content-Type: text/plain;
charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Subject: =?utf-8?B?emHFvMOzxYLEhw==?=
Message-ID:
Date: Fri, 27 Sep 2013 17:59:48 +0200

Hello
Mail.dll S: 6d9d6c00b9294efb OK APPEND completed
Mail.dll C: 0a55ca5702664b94 UID FETCH 100065 (UID RFC822.SIZE INTERNALDATE FLAGS ENVELOPE BODYSTRUCTURE)
Mail.dll S: * 3 FETCH (UID 100065 RFC822.SIZE 290 FLAGS (\Seen) INTERNALDATE "27-Sep-2013 15:59:44 +0000" ENVELOPE ("Fri, 27 Sep 2013 17:59:48 +0200" "za????" NIL NIL NIL NIL NIL NIL NIL "") BODYSTRUCTURE ("TEXT" "plain" ("charset" "utf-8") NIL NIL "7BIT" 5 1 NIL NIL NIL NIL))
Mail.dll S: 0a55ca5702664b94 OK FETCH Completed


Workaround

You can workaround this issue, requesting subject header explicitly or downloading all headers (using GetHeadersByUID):

var eml = client.GetSpecificHeadersByUID(uploaded, new[]{"subject"}).EmlData;
string subject = new MailBuilder().CreateFromEml(eml).Subject;

Although slower, correct data is returned:


66496df367914917 UID FETCH 100078 (UID BODY[HEADER.FIELDS (SUBJECT)])
Mail.dll: 13 19:38:29 S: * 7 FETCH (UID 100078 BODY[HEADER.FIELDS ("SUBJECT")] {41}
Mail.dll: 13 19:38:30 S: Subject: =?utf-8?B?emHFvMOzxYLEhw==?=

Mail.dll: 13 19:38:30 S: )
Mail.dll: 13 19:38:30 S: 66496df367914917 OK FETCH Completed