This is not a download error.
In your code you must be casting object of type MailAddress to MailBox explicitly.
MailAddress can be MailBox -or- MailGroup (both inherit from MailAddress).
Please use GetMailboxes method:
var mailboxes = email.To.SelectMany(address => address.GetMailboxes())
-or-
handle mailboxes and groups differently.
You can find more on this here:
https://www.limilabs.com/blog/how-to-access-to-cc-bcc-fields