Get new emails using IMAP

This article describes how to receive new email messages using Mail.dll .NET IMAP library.

In many situations you can relay on IMAP server to keep track which messages are unseen and just download unseen messages from IMAP server. However when user interaction or other program connecting and downloading emails may remove UNSEEN flag from the message, you may be forced to manually track which messages where already downloaded and which are new.

You could of course download all Unique IDs (UIDs) of the messages that are in the mailbox, but there is a more elegant way.

Unique IDs (UIDs) in IMAP have some very interesting feature – they are assigned in incremental order. This means that new messages always have higher UIDs than old ones.

This is the reason you only need to remember the newest (greatest) UID that was downloaded during a previous session (connection to the IMAP server). On the next session you need to search IMAP for UIDs that are greater than the one remembered.

First we need to load largest uid from the previous session and connect to the IMAP server. In this exmaple we select INBOX, but you can use Select method to select a different folder.

// C#

long? largestUID = LoadLargestFromPreviousRun();

using (Imap imap = new Imap ())
{
    imap.Connect("imap.example.com");  // or ConnectSSL for SSL
    imap.UseBestLogin("user", "password");

    imap.SelectInbox();
' VB.NET

Dim largestUID As System.Nullable(Of Long) = LoadLargestFromPreviousRun()

Using imap As New Imap ()
    imap.Connect("imap.example.com")   ' or ConnectSSL for SSL
    imap.UseBestLogin("user", "password")

    imap.SelectInbox()

Next step is to select folder, download unique ids (UIDs) larger than largestUID value that was stored on the previous run. This way we obtain uids of the new emails.

// C#

List<long> uids;
if (largestUID == null)
{
    uids = imap.GetAll();
}
else
{
    uids = imap.Search().Where(
        Expression.UID(Range.From(largestUID.Value)));
    uids.Remove(largestUID);
}

' VB.NET

Dim uids As List(Of Long)
If largestUID Is Nothing Then
	uids = imap.GetAll()
Else
	uids = imap.Search().Where(Expression.UID(Range.From(largestUID.Value)))
	uids.Remove(largestUID)
End If

You should note uids.Remove in the code above. It is used because Range.From creates a range of uids greater or equal to the specified value and * represents the largest value in the mailbox, so the largest uid is always included.

Finally we’ll iterate through the UID list and use GetMessageByUID method to download each message from the server. You can use MailBuilder class to parse those email messages, extract attachments and process them anyway you like. The last step is to save the last UID that was processed:

// C#

foreach (long uid in uids)
{
    var eml = imap.GetMessageByUID(uid);
    IMail email = new MailBuilder().CreateFromEml(eml);

    Console.WriteLine(email.Subject);
    Console.WriteLine(email.Text);
   
    SaveLargestUID(uid.Value);
}
' VB.NET

For Each uid As Long In uids
	Dim eml = imap.GetMessageByUID(uid)
	Dim email As IMail = New MailBuilder().CreateFromEml(eml)

	Console.WriteLine(email.Subject)
	Console.WriteLine(email.Text)

	SaveLargestUID(uid.Value)
Next

SaveLargestUID, LoadLargest… methods

You need to write those methods on your own. They should save/load largest uid from/to your preferred storage (such as database, file, registry).

// C#

private void SaveLargestUID(long uid)
{
    // Your code that saves the uid.
}

private long  LoadLargestFromPreviousRun()
{
    // Your code that loads the largest uid (null on the first run).
}
' VB.NET

Private Sub SaveLargestUID(uid As Long)
    ' Your code that saves the uid.
End Sub

Private Function LoadLargestFromPreviousRun() As Long
    ' Your code that loads the largest uid (null on the first run).
End Sub

UIDs across sessions – UIDValidity

It is advised for IMAP servers to not change UIDs between sessions. There are however situations when server may invalidate all UIDs that were in the mailbox previously.

Consider a user that deletes a folder (mailbox) including all its messages and creates new one with exactly the same name. All uids that were downloaded previously are no longer valid – they simply don’t exist in this folder anymore.

You can recognize such situation by comparing FolderStatus.UIDValidity value. If the FolderStatus.UIDValidity number hasn’t changed, then the UIDs are still valid. Below is the sample showing how to print UIDValidity:

// C#

FolderStatus status = client.SelectInbox();
Console.WriteLine(status.UIDValidity);

' VB.NET

Dim status As FolderStatus = client.SelectInbox()
Console.WriteLine(status.UIDValidity)

Entire samples

Below you can find full samples in both C# and VB.NET:

// C#

using Limilabs.Mail;
using Limilabs.Client.IMAP;

internal class LastRun
{
    public long UIDValidity { get; set; }
    public long LargestUID { get; set; }
}

private static void Main(string[] args)
{
    LastRun last = LoadPreviousRun();

    using (Imap imap = new Imap())
    {
        imap.Connect("imap.example.com"); // or ConnectSSL for SSL
        imap.UseBestLogin("user", "password");

        FolderStatus status = imap.SelectInbox();

        List<long> uids;
        if (last == null || last.UIDValidity != status.UIDValidity)
        {
            uids = imap.GetAll();
        }
        else
        {
            uids = imap.Search().Where(
                Expression.UID(Range.From(last.LargestUID)));
            uids.Remove(last.LargestUID);
        }

        foreach (long uid in uids)
        {
            var eml = imap.GetMessageByUID(uid);
            IMail email = new MailBuilder()
                .CreateFromEml(eml);

            Console.WriteLine(email.Subject);
            Console.WriteLine(email.Text);

            LastRun current = new LastRun
                                  {
                                      UIDValidity = status.UIDValidity, 
                                      LargestUID = uid
                                  };
            SaveThisRun(current);
        }
        imap.Close();
    }
}
' VB.NET

Imports Limilabs.Mail
Imports Limilabs.Client.IMAP

Friend Class LastRun
	Public Property UIDValidity () As Long
		Get
			Return m_UIDValidity 
		End Get
		Set
			m_UIDValidity = Value
		End Set
	End Property
	Private m_UIDValidity As Long
	Public Property LargestUID() As Long
		Get
			Return m_LargestUID
		End Get
		Set
			m_LargestUID = Value
		End Set
	End Property
	Private m_LargestUID As Long
End Class


Private Shared Sub Main(args As String())
    Dim last As LastRun = LoadPreviousRun()

    Using imap As New Imap()
        imap.Connect("imap.example.com")		' or ConnectSSL for SSL
        imap.UseBestLogin("user", "password")

        Dim status As FolderStatus = imap.SelectInbox()

        If last Is Nothing OrElse last.UIDValidity <> status.UIDValidity Then
	        uids = imap.GetAll()
        Else
	        uids = imap.Search().Where(Expression.UID(Range.From(last.LargestUID)))
        	uids.Remove(last.LargestUID)
        End If

        For Each uid As Long In uids
            Dim eml = imap.GetMessageByUID(uid)
            Dim email As IMail = New MailBuilder().CreateFromEml(eml)

            Console.WriteLine(email.Subject)
            Console.WriteLine(email.Text)

            Dim current As New LastRun() With { _
                .UIDValidity = status.UIDValidity, _
                .LargestUID = uid _
            }
            SaveThisRun(current)
        Next
        imap.Close()
    End Using
End Sub

SaveThisRun, LoadPreviousRun methods

You need to write those methods on your own. They should save/load last run data from/to your preferred storage (such as database, file, registry).

// C#

private void SaveThisRun(LastRun run)
{
    // Your code that saves run data.
}

private LastRun LoadPreviousRun()
{
    // Your code that loads last run data (null on the first run).
}
' VB.NET

Private Sub SaveThisRun(uid As LastRun )
    ' Your code that saves run data.
End Sub

Private Function LoadPreviousRun() As LastRun
    ' Your code that loads last run data (null on the first run).
End Function

Thread emails using IMAP

The THREAD extension to the IMAP protocol provide a means of server-based threading of messages, without requiring that the IMAP component download the necessary data to do so itself.

Following sample shows how to use Imap Thread method to thread all emails available in the mailbox.

  • First it checks if THREAD extension is available on the server (defined in RFC 5256).
  • It also downloads envelopes of all email messages to get messages subjects and basic data.
  • Finally recursive ShowThread method is used. It displays message UID and subject in a formatted tree manner.

There is one interesting thing in this sample: you can use any search criterion to get messages from the IMAP server. For example you could only get unseen messages or messages that are unseen and contain a certain word.

// C#

using (Imap imap = new Imap())
{
    imap.Connect("imap.example.org");    // or ConnectSSL
    imap.UseBestLogin("user", "password");
    
    ThreadMethod method = ThreadMethod.ChooseBest(imap.SupportedThreadMethods());
    if (method == null)
        throw new Exception("This server does not support any threading algorithm.");

    imap.SelectInbox();

    List<MessageThread> threads = imap.Thread(method).Where(Expression.All());
    List<Envelope> envelopes = imap.GetEnvelopeByUID(
        imap.Search().Where(Expression.All()));

    foreach (MessageThread thread in threads)
    {
        ShowThread(0, thread, envelopes);
    }
    imap.Close();
}

public void ShowThread(int level, MessageThread thread, List<Envelope> envelopes)
{
    string indent = new string(' ', level*4);
    string subject = envelopes.Find(x => x.UID == thread.UID).Subject;
    Console.WriteLine("{0}{1}: {2}", indent, thread.UID, subject);

    foreach (MessageThread child in thread.Children)
    {
        ShowThread(level + 1, child, envelopes);
    }
}
' VB.NET

Using imap As New Imap()
	imap.UseBestLogin("imap.example.org")	' or ConnectSSL
	imap.Login("user", "password")

	Dim method As ThreadMethod = ThreadMethod.ChooseBest(imap.SupportedThreadMethods())
	If method Is Nothing Then
		Throw New Exception("This server does not support any threading algorithm.")
	End If

	imap.SelectInbox()

	Dim threads As List(Of MessageThread) = _ 
		imap.Thread(method).Where(Expression.All())
	Dim envelopes As List(Of Envelope) = 
		_ imap.GetEnvelopeByUID(imap.Search().Where(Expression.All()))

	For Each thread As MessageThread In threads
		ShowThread(0, thread, envelopes)
	Next
	imap.Close()
End Using

Public Sub ShowThread(level As Integer, thread As MessageThread, envelopes As List(Of Envelope))
	Dim indent As New String(" "C, level * 4)
	Dim subject As String = envelopes.Find(Function(x) x.UID = thread.UID).Subject
	Console.WriteLine("{0}{1}: {2}", indent, thread.UID, subject)

	For Each child As MessageThread In thread.Children
		ShowThread(level + 1, child, envelopes)
	Next
End Sub

Here you can see the results:

2299: First thread
    2300: Re: First thread
    2301: Re: First thread
2302: Second thread
    2303: Re: Second thread
    2304: Re: Re: Second thread
    2305: Re: Re: Re: Second thread

FTP uploading files using patterns

Uploading files using patters is a unique feature of our FTP component. It allows you fast upload of files of certain types.

Here’s the sample that uploads all text files (*.txt) files from C drive, to newly created ‘Uploads’ folder on FTP server. The search includes all child folders recursively – note the true parameter in LocalSearchOptions constructor. Ftp component is going to create all necessary folders on the remote FTP server for you.

Upload using wildcard pattern

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL

    ftp.CreateFolder("Uploads");

    LocalSearchOptions options = new LocalSearchOptions("*.txt", true);
    ftp.UploadFiles("Uploads", @"c:\", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL
    ftp.CreateFolder("Uploads")

    Dim options As New LocalSearchOptions("*.txt", True)
    ftp.UploadFiles("Uploads", "c:\", options)

    ftp.Close()
End Using

Upload using regex pattern

You can also use LocalSearchOptions.UseRegexMatch method, if you want to use regex patterns:

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL

    ftp.CreateFolder("Uploads");

    LocalSearchOptions options = new LocalSearchOptions();
    options.UseRegexMatch(@"^.*$", @"^.*\.txt$", true);
    ftp.UploadFiles("Uploads", @"c:\", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL

    ftp.CreateFolder("Uploads")

    Dim options As New LocalSearchOptions()
    options.UseRegexMatch("^.*$", "^.*\.txt$", True)
    ftp.UploadFiles("Uploads", "c:\", options)

    ftp.Close()
End Using

FTP downloading files using patterns

Downloading files using patters is a unique feature of our FTP .NET component. It allows you fast download of files of certain types.

Here’s the sample that download all text files (*.txt) files from remote Uploads folder to Downloads folder located on C drive. Remote search includes all child folders recursively – note the true parameter in RemoteSearchOptions constructor. Ftp component is going to create all necessary folders on the local computer.

Download using wildcard pattern

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL
    
    Directory.CreateDirectory(@"c:\Downloads");

    RemoteSearchOptions options = new RemoteSearchOptions("*.txt", true);
    ftp.DownloadFiles("Uploads", @"c:\Downloads", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")    ' or ConnectSSL

    Directory.CreateDirectory("c:\Downloads")
    
    Dim options As RemoteSearchOptions = _
        New RemoteSearchOptions("*.txt", True)
    ftp.DownloadFiles("Uploads", "c:\Downloads", options)

    ftp.Close()
End Using

Download using regex pattern

You can also use RemoteSearchOptions.UseRegexMatch method, if you want to use regex patterns on remote names:

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL
    
    Directory.CreateDirectory(@"c:\Downloads");

    RemoteSearchOptions options = new RemoteSearchOptions();
    options.UseRegexMatch(@"^.*$", @"^.*\.txt$", true);
    ftp.DownloadFiles("Uploads", @"c:\Downloads", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL

    Directory.CreateDirectory("c:\Downloads")

    Dim options As RemoteSearchOptions = _
        New RemoteSearchOptions()
    options.UseRegexMatch("^.*$", "^.*\.txt$", True)
    ftp.DownloadFiles("Uploads", "c:\Downloads", options)

    ftp.Close()
End Using

Ftp zlib compression

Ftp.dll FTP component supports zlib compression for all data transfers.

What is zlib

zlib is a compression method which can greatly decrease the size of data, similar to popular compression format – Zip.

How to use zlib

zlib can be used with Ftp.dll FTP/FTPS component thanks to the newly implemented MODE Z command. This command and format have been already adopted by popular FTP servers.

Ftp.dll automatically turns compression on, if it is supported by FTP server. zlib compression is transparent to users. It is activeted using MODE Z command. Once sent, client sends and receives compressed data, so all directory listings and file exchanges between server and client will be compressed.

What is the benefit?

Faster data transfers! Directory listing, which is text, can be highly compressed with zlib thus boosting the server and client network speed and reactivity.

Transfers of html, scripts or large logfiles (which are text) no longer needs to be zipped before being sent via ftp and should generally experience a 3-4 times gain in data transfers.

For example, a 60MB log file can turn into a 5MB data exchange when transferred with MODE Z enabled.

Depending on the file content, you will see different results:

  • Text files : ~15-20% of original size.
  • Html files : ~25-30% of original size.
  • Media, video, sound : ~90-95% of original size.
  • Already compressed documents with Zip, Rar, Ace etc. will see almost no gain at all.