Yahoo SMTP 8 BITMIME bug

It’s sometimes hard to believe, that even the biggest make such mistakes.

Here’s the capability (EHLO) response from Yahoo’s SMTP server:

S: 220 smtp.mail.yahoo.com ESMTP ready
C: EHLO [192.168.0.11]
S: 250-smtp.mail.yahoo.com
S: 250-PIPELINING
S: 250-SIZE 41697280
S: 250-8 BITMIME
S: 250 STARTTLS

Can you spot the problem? It’s not that easy, and it has been brought to my attention by one of our customers:

S: 220 smtp.mail.yahoo.com ESMTP ready
C: EHLO [192.168.0.11]
S: 250-smtp.mail.yahoo.com
S: 250-PIPELINING
S: 250-SIZE 41697280
S: 250-8 BITMIME
S: 250 STARTTLS

It supposed to be: 8BITMIME, at least according to RFC6152

the EHLO keyword value associated with the extension is 8BITMIME;

Most annoying part is that in many cases, space is used to split extensions keyword and its parameters e.g. SIZE 41697280. We want Mail.dll to be robust enough to accept custom extensions or extensions unknown yet, that follow this standard.

Shame on you Yahoo!.

Joining message/partial emails

With Mail.dll you can join a message split across multiple message/partial emails.

“Message/partial” emails allow large objects to be delivered as several mail messages and automatically joined by the receiving client. This mechanism can be used when intermediate transport agents, such as SMTP servers, are limiting the size of individual mails that can be sent. Content-Type “message/partial” indicates that the body contains a fragment of a larger email.

// C#

IMail part1 = ...
IMail part2 = ...

PartialMailJoiner joiner = new PartialMailJoiner();
joiner.Add(part1);
joiner.Add(part2);

IMail email = joiner.Join();
var attachments = email.Attachments;
' VB.NET

Dim part1 As IMail = "..."
Dim part2 As IMail = "..."

Dim joiner As New PartialMailJoiner()
joiner.Add(part1)
joiner.Add(part2)

Dim email As IMail = joiner.Join()
Dim attachments = email.Attachments

PartialMailJoiner.Add method checks if all parts of message/partial email are present and you can use PartialMailJoiner.Join method:

// C#

IMail part1 = ...
IMail part2 = ...

List<IMail> parts = new List<IMail>{part1, part2};

PartialMailJoiner joiner = new PartialMailJoiner();

foreach (IMail part in parts)
{
    if (part.IsPartial)
    {
        bool allPartsPresent = joiner.Add(part);

        if (allPartsPresent)
        {
            IMail email = joiner.Join();
            var attachments = email.Attachments;
        }
    }
}

' VB.NET

Dim part1 As IMail = ...
Dim part2 As IMail = ...

Dim parts As New List(Of IMail)() From { _
	part1, _
	part2 _
}

Dim joiner As New PartialMailJoiner()

For Each part As IMail In parts
    If part.IsPartial Then
        Dim allPartsPresent As Boolean = joiner.Add(part)

	If allPartsPresent Then
            Dim email As IMail = joiner.Join()
	    Dim attachments = email.Attachments
        End If
    End If
Next

Joining fragmented message is a bit tricky. Messages filtered using Content-Type’s id property and then sorted by Content-Type’s number property. Note that part numbering begins with 1, not 0.

The headers of the encapsulated message must be merged with the headers of the enclosing entities. Following rules must be used:

  • All of the header fields from the initial enclosing message, except those that start with “Content-” and the specific header fields” “Subject”, “Message-ID”, “Encrypted”, and “MIME-Version”, must be copied, in order, to the new message.
  • The header fields in the enclosed message which start with “Content-“, plus the “Subject”, “Message-ID”, “Encrypted”, and “MIME-Version” fields, must be appended, in order, to the header fields of the new message. all others will be ignored and dropped.
  • All of the header fields from the second and any subsequent enclosing messages are ignored.

Using Limilabs’ Ftp.dll with zOS Mainframes

I purchased the Limilabs FTP product for FTP because I needed to send data to and from an IBM mainframe from my VB.NET program running in Windows. In particular I needed to be able to submit jobs, and receive the job output. These notes show how it’s done.

Introduction

FTP to/from IBM computers is pretty much like any other FTP except for two things.
1. IBM Mainframe and Midrange computers mostly use EBCDIC encoding rather than ASCII. With the FTP defaults your data can be garbled and useless when it arrives at the other end.
2. The SITE command is used to submit jobs to the mainframe, and get the results back.

Getting Started

I created a class called “JazzFTP” to wrap the Limilabs’ code. This was going to contain the functions that I wanted for my project, and so I started by defining the common elements that all methods would use. In my situation every FTP would be authenticated, and would be exchanging text data (not binary) with the remote computer.

Here is the initial class definition:

Imports Limilabs.FTP.Client

Public Class JazzFTP
    '   This class wraps the FTP library from Limilabs (/ftp)
    '   All methods
    '   1   Connect and logon using information from MySettings:  Sub LoginFTP
    '   2   Perform their action based on their parameters
    '   3   Close the connection
    Dim ftp As New Ftp()
    Dim response As FtpResponse
    Private Sub LoginFTP()
        ftp.Connect(My.Settings.SubmitIP)
        ftp.Login(My.Settings.Userid, My.Settings.Password)
    End Sub
    '   My functions will be written here
End Class

Basic FTP

Here is my first method, a basic function to upload a text file: –

    Function Upload(DestinationFile As String, Uploadfile As String) As String 
        LoginFTP()
        ftp.TransfersDataType = FtpDataType.Ascii
        response = ftp.Upload(DestinationFile, Uploadfile)
        ftp.Close()
        Return response.message
    End Function

FTP’s default is Binary, which is correct if you are transmitting a .JPG or other binary object, and it probably doesn’t matter if you are transmitting text to/from another Windows computer or a Unix computer. However if you are transmitting to/from an IBM mainframe or midrange computer it probably needs EBCDIC rather than ASCII characters. You must tell it that the file is Ascii text, not binary, otherwise it won’t be converted and it will be gibberish when you examine it on the mainframe.

Although this code above works, it is very fragile: the FTP server has to be up and running, you have to get the connection details exactly right, the source and destination files must exist, and so on. Since I couldn’t guarantee all of these details, I enclosed the code in Try/Catch to deal with any errors. For the time being I’ve simply used MsgBox to display the error message.

    Function Upload(DestinationFile As String, Uploadfile As String) As String
        Try
            LoginFTP()
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload(DestinationFile, Uploadfile)
            ftp.Close()
            Return response.EndLine
        Catch ex As Exception
            MsgBox(ex.Message)
            Return ex.Message
        End Try
    End Function

Download, which is not illustrated, is similar except that you’d use ftp.Download. Again, you specify:

            ftp.TransfersDataType = FtpDataType.Ascii

Submitting Jobs and Receiving Job Output

Submitting a job is essentially an upload with a twist. Instead of uploading the file containing the job to a named file on the mainframe, you upload it to the JES (Job Entry System) input queue. Here is the basic code:

            response = ftp.Site("FILETYPE=JES")
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload("JES", JCLFile)

The first line uses “ftp.Site”. Site means that this is a site-specific command, something that the FTP system at the other end will presumably know about. For an IBM mainframe “FILETYPE=JES” means that the data is going to and from JES.

The third line uploads the file in which we have prepared our job: in this case JCLFile is a file containing something like this: –

//IBMUSERH JOB  ,CLASS=A,MSGCLASS=H,NOTIFY=&SYSUID,COND=(8,LT) 
//*** COPY SOURCE INTO SOURCE LIBRARY
//COPY EXEC PGM=IEBGENER
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT2 DD DSN=IBMUSER.MANAJAZZ.SRCLIB(CRDTA1),DISP=SHR
//SYSUT1 DD *

Like any upload the FTP client syntax requires a destination file name, but because of the preceding FILETYPE=JES this will be ignored. I have written “JES” purely for documentation.

This will submit the job and it will run, appearing in the output like this:

tasks

Of course we can view the job output on the mainframe, but we may want to return it to Windows. We do this by downloading the file by naming the JobID – JOB00594 in this case – and again using the Site command. The essential code is: –

        response = ftp.Site("FILETYPE=JES")
        ftp.TransfersDataType = FtpDataType.Ascii
        ftp.Download(Jobname, LocalPath)

But how do we get the Jobname? JES returns a message with this information when the job is submitted. We need to add code to the Job Submission logic to extract this from the message. In function JobSub

       response = ftp.Upload("JES", JCLFile)

upload the job and (if all goes well) returns a message like
“It is known to JES as JOB00594″
This code extracts JOB00594” and puts it into variable JobName

       Dim TestString As String = "It is known to JES as"
       If Mid(response.Message, 1, Len(TestString)) = TestString Then  'Should be true
           Jobname = Trim(Mid(response.Message, Len(TestString) + 1))
       End If

Now, since it is logical that if we submit a job we’ll want to get it back, I coded this in the JobSub function: –

    Function JobSub(JCLFile As String, ByRef Jobname As String) As FtpResponse
        '   JCLFile is path to a .JCL file (format .txt) containing the job to be submitted
        '   If successful submission, the job name is returned in JobName
        Dim Tstring As String = ""
        Try
            Jobname = "Unknown"
            LoginFTP()
            response = ftp.Site("FILETYPE=JES")
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload("JES", JCLFile)
            Dim TestString As String = "It is known to JES as"
            If Mid(response.Message, 1, Len(TestString)) = TestString Then  'Should be true
                Jobname = Trim(Mid(response.Message, Len(TestString) + 1))
            End If
            JobGet(Jobname, False)
            ftp.Close()
        Catch ex As Exception
            MsgBox(ex.Message, MsgBoxStyle.OkOnly, "Jazz FTP")
            Return response
        End Try
        Return response
    End Function

and I coded JobGet to accept these parameters: –

    Function JobGet(Jobname As String, Optional Login As Boolean = True) As FtpResponse
        '   Get job output, save as Jobname.txt in Jazz Program Library.  
        If Login Then
            LoginFTP()
        End If
        response = ftp.Site("FILETYPE=JES")
        ftp.TransfersDataType = FtpDataType.Ascii
        Dim LocalPath As String = My.Settings.UserCommonPath & "\" & My.Settings.Programs & "\" & Jobname & ".txt"
        Jazzworkbench.ShowBtnResults(Jobname, Jazzworkbench.ResultsStatus.Pending)
        ftp.Download(Jobname, LocalPath)
        ftp.DeleteFile(Jobname)
        Jazzworkbench.ShowBtnResults(Jobname, Jazzworkbench.ResultsStatus.JobReturned)
        ftp.Close()
        Return response
    End Function

Once the job output has been downloaded I didn’t want to leave it cluttering up my Held Job Output Queue, so after the Download

        ftp.DeleteFile(Jobname)

gets rid of it.

This all works for my test jobs (which are very quick) and provided that mainframe FTP server is available.

If you want to know any more about my project to revolutionize mainframe programming, then have a look at www.jazzsoftware.co.nz

Best wishes with your programming,
Robert Barnes.

Processing applesingle and appledouble email attachments

Mail.dll now supports applesingle and appledouble email attachments.

AppleSingle and AppleDouble are file formats developed to store Mac OS “dual-forked” files (data fork and resource fork)

AppleSingle combines both file forks and the related Finder meta-file information into a single file MIME entity, usually with application/applesingle content-type email header. Applesingle files are parsed by Mail.dll and data fork is extracted as a standard email attachment available through IMail.Attachments collection.

AppleDouble stores both forks as two separate files. Thus it uses multipart mime type – inside two MIME entities are stored: application/applefile, which contains resource fork only, and regular MIME attachment e.g. application/octet-stream content-type.

In both cases you simply use IMail.Attachments collection to access attachments.

byte[] eml = ...;

IMail mail = new MailBuilder().CreateFromEml(eml);
Assert.AreEqual(1, mail.Attachments.Count);
MimeData att = (MimeData) mail.Attachments[0];

byte[] data = att.Data; // data fork

You can control, if apple attachments should be processed using MailBuilder.ProcessAppleAutomatically property. When it is set to false applesingle files are not parsed and data fork is not extracted.

byte[] eml = ...;

MailBuilder builder = new MailBuilder();
builder.ProcessBinHexAutomatically = false;
IMail mail = builder.CreateFromEml(eml);

MimeAppleSingle appleSingle = (MimeAppleSingle) mail.Attachments[0];

Assert.AreEqual(ContentType.ApplicationAppleSingle, appleSingle.ContentType);

byte[] resource = appleSingle.AppleSingle.Resources;
string name = appleSingle.AppleSingle.RealName;

OAuth 2.0 with Gmail over IMAP for service account

You can also read how to use:

 

In this article I’ll show how to access Gmail account of any domain user, using OAuth 2.0, .NET IMAP component and service accounts. The basic idea is that domain administrator can use this method to access user email without knowing user’s password.

This scenario is very similar to 2-legged OAuth, which uses OAuth 1.0a. Although it still works, it has been deprecated by Google and OAuth 2.0 service accounts were introduced.

The following describes how to use XOAUTH2 and OAuth 2.0 to achieve the equivalent of 2-legged OAuth.

Google.Apis

Use Nuget to download “Google.Apis.Auth” package.

Import namespaces:

// c#

using Google.Apis.Auth.OAuth2;
using Google.Apis.Auth.OAuth2.Flows;
using Google.Apis.Auth.OAuth2.Requests;
using Google.Apis.Auth.OAuth2.Responses;

using System.Security.Cryptography.X509Certificates;

using Limilabs.Client.Authentication.Google;

using Limilabs.Client.IMAP;
' VB.NET 

Imports Google.Apis.Auth.OAuth2
Imports Google.Apis.Auth.OAuth2.Flows
Imports Google.Apis.Auth.OAuth2.Requests
Imports Google.Apis.Auth.OAuth2.Responses

Imports System.Security.Cryptography.X509Certificates

Imports Limilabs.Client.Authentication.Google

Imports Limilabs.Client.IMAP

Google Cloud

First you need to visit Google Cloud Console and create a project:

Now create a new service account:

Add a service name and remember an email address assigned to your service:

Then you need to create a private key for this service:

Download and save this private key (XYZ.p12), you’ll need that later:

Google Domain administration

Final part is to allow this service to access your domain. You’ll perform this steps in your domain administration panel.

Remember the Client ID first, and go to your domain administration panel:

In the main menu select Security / Access and data control / API controls

Then Manage domain wide delegation

Use previously remembered Client ID and https://mail.google.com/, which is IMAP/SMTP API scope:

Alternatively you can use https://www.googleapis.com/auth/gmail.imap_admin scope.

When authorized with this scope, IMAP connections behave differently:

  • All labels are shown via IMAP, even if users disabled “Show in IMAP” for the label in the Gmail settings.
  • All messages are shown via IMAP, regardless of what the user set in “Folder Size Limits” in the Gmail settings.

Access IMAP/SMTP server

// C#

const string serviceAccountEmail = "name@xxxxxxxxxx.gserviceaccount.com";
const string serviceAccountCertPath = @"c:\XYZ.p12";
const string serviceAccountCertPassword = "notasecret";
const string userEmail = "user@your-domain.com";

X509Certificate2 certificate = new X509Certificate2(
    serviceAccountCertPath,
    serviceAccountCertPassword,
    X509KeyStorageFlags.Exportable);

ServiceAccountCredential credential = new ServiceAccountCredential(
    new ServiceAccountCredential.Initializer(serviceAccountEmail)
    {
        Scopes = new[] { "https://mail.google.com/" },
        // Scopes = new[] { "https://www.googleapis.com/auth/gmail.imap_admin" },

        User = userEmail
    }.FromCertificate(certificate));

bool success = await credential.RequestAccessTokenAsync(
    CancellationToken.None);

using (Imap imap = new Imap())
{
    imap.ConnectSSL("imap.gmail.com");
    imap.LoginOAUTH2(userEmail, credential.Token.AccessToken);

    imap.SelectInbox();

    foreach (long uid in uids)
    {
        var eml = client.GetMessageByUID(uid);
        IMail email = new MailBuilder().CreateFromEml(eml);
        Console.WriteLine(email.Subject);
    }

    imap.Close();
}
' VB.NET

Const serviceAccountEmail As String = "name@xxxxxxxxxx.gserviceaccount.com"
Const serviceAccountCertPath As String = "c:\XYZ.p12"
Const serviceAccountCertPassword As String = "notasecret"
Const userEmail As String = "user@your-domain.com"

Dim certificate As New X509Certificate2(serviceAccountCertPath, serviceAccountCertPassword, X509KeyStorageFlags.Exportable)

Dim credential As New ServiceAccountCredential(New ServiceAccountCredential.Initializer(serviceAccountEmail) With { _
	.Scopes = {"https://mail.google.com/"}, _
	' .Scopes = {"https://www.googleapis.com/auth/gmail.imap_admin"}, _
	.User = userEmail _
}.FromCertificate(certificate))

Dim success As Boolean = credential.RequestAccessTokenAsync(
    CancellationToken.None).Result

Using imap As New Imap()
	imap.ConnectSSL("imap.gmail.com")
	imap.LoginOAUTH2(userEmail, credential.Token.AccessToken)

	imap.SelectInbox()

	For Each uid As Long In uids
		Dim eml = client.GetMessageByUID(uid)
		Dim email As IMail = New MailBuilder().CreateFromEml(eml)
		Console.WriteLine(email.Subject)
	Next

	imap.Close()
End Using