Migrating email providers (Office 365 to Mythic Beasts)

29th June 2024 by Genevieve Clifford

Recently I’ve done lots of life admin and sysadmin work. On the latter part, I’ve started hosting some cloud tools that are useful for my PhD work, these include: HedgeDoc (collaborative note-taking), Vikunja (planning / Kanban board), Rallly (for time polls), and Miniflux (feed reader). I really care about free and open-source software (as I’ve discussed briefly previously), and have committed to using FOSS in my PhD work and personal life except where absolutely unavoidable. Also, much of the impetus in hosting my own infrastructure comes from Elmar mc.fly Lecher’s “Run your own fucking infrastructure - 2024 edition” talk¹ at EMF 2024 (unfortunately not recorded). This has culminated in me having a very sysadmin-focussed mindset lately.

On the former point of life admin (this is related), I had moved one of my current accounts, given my former bank’s (Barclay’s) role in financing the ongoing genocide of the Palestinian people. So far, this has had only one side effect: it caused a failed payment for my Office 365 subscription.

About five years ago during my final year of my undergraduate degree (this makes me feel old), I had set up a domain and web and email hosting to go with it. I am not entirely sure why I chose Office 365, I can only speculate because I used it at university, and it seemed interesting? Either way, I don’t really use this domain or email address any more, and the failed payment led me to ask why I was still using Microsoft’s proprietary groupware platform, when I already use FOSS alternatives, and have a low regard for Microsoft generally.

I had previously thought of moving providers, but decided against it given a lack of spoons at the time, and limited availability of documentation². However, I have a plan and time now to undertake this switch. The plan for this blog post is as follows: I will detail the approach I am going to take, and then my experience with doing that.

Expected workflow

Much of this follows my work in setting up aerc and simonrob/email-oauth2-proxy³ to work with each other. I am also inspired by Drew DeVault’s approach⁴ to using aerc with Postfix and isync / mbsync for offline email syncing using Maildirs. I expect this to proceed as follows:

1. Use email-oauth2-proxy to proxy a local IMAP connection to my O365 instance
2. Set up mbsync to copy emails from the “remote” IMAP mailbox to a local Maildir
3. Check the integrity of downloaded emails (i.e., doing things like removing the annoying Safelinks Protection using an approach like francisyan/safelinks-decoder, but rewritten in Python and running locally as a script).
4. Remove spam.
5. Switch DNS to new provider, then either:
   1. Copy generated Maildirs to new mailbox using scp or similar
   2. Just maintain a local backup of the emails if access via IMAP isn’t actually needed (thanks Kim for convincing me I might not need this)

Actual workflow

Stage 1: emailproxy setup

I started by installed emailproxy by using pipx (which I installed using asdf) with pipx install emailproxy\[gui\] (formatted like this because of zsh). I then edited my previous emailproxy.conf file and saved it in the home directory, its contents are:

[Server setup]

[IMAP-1993]
server_address = outlook.office365.com
server_port = 993
local_address = 127.0.0.1

[SMTP-1587]
server_address = smtp.office365.com
server_port = 587
server_starttls = True
local_address = 127.0.0.1

[Account setup]

[user@example.com]
permission_url = https://login.microsoftonline.com/common/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/common/oauth2/v2.0/token
oauth2_scope = https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/POP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access
redirect_uri = http://localhost
client_id = 08162f7c-0fd2-4200-a84a-f25a4db0b584
client_secret = TxRBilcHdC6WGBee]fs?QR:SJ8nI[g82

Of course, my email address is not actually user@example.com, replace as necessary. Eagle-eyed viewers may have spotted client_id andclient_secret in the above, these are the settings for Mozilla Thunderbird. If you have admin permissions for your O365 instance, you can generate a new app, if not, there are several linked in Simon’s README.md document for the proxy.

This is unnecessary, but I installed aerc to validate that I’d set up emailproxy correctly. On MacOS, aerc saves its configuration in ~/Library/Preferences/aerc/accounts.conf (on Linux this is in ~/.config/aerc/accounts.conf IIRC). If this is useful to you, the setup I performed consists of:

[account name]
source        = imap+insecure://user%40example.com:1234@localhost:1993
outgoing      = smtp+insecure://user%40example.com:1234@localhost:1587
default       = INBOX
from          = "Example User" <user@example.com>
cache-headers = true

I then launched emailproxy with emailproxy --no-gui --external-auth (you may not need to do this), but I then copied the top link in my terminal into my browser where I’m logged into my O365 account, authorised “Mozilla Thunderbird” to have OAuth2.0 access to my O365 account and then pasted the link from the address bar into the terminal. And waiting a few seconds longer for aerc to try connecting again, it worked! I could see all of my emails from my O365 account in aerc! Of course, you don’t actually need aerc to start the OAuth2.0 workflow, there’s some information in README.md about using telnet instead to log into the IMAP server (CMD + F a1 login e@mail.com to find it).

Stage 2: mbsync setup

I installed mbsync by using Homebrew on my Mac (brew install isync), I had no idea where the config file referenced in the previous blog post was stored, so I ran mbsync with no arguments; turns out it’s expecting to find one at ~/.mbsyncrc, so I made one there. I then basically copied the configuration file, but made a few changes, I show below my version of this, with a diff underneath from the provided version.

IMAPAccount example
Host localhost
Port 1993
User user@example.com
Pass 1234
SSLType None

MaildirStore local
Path ~/mail/
INBOX ~/mail/INBOX
SubFolders Verbatim

IMAPStore example
Account example

Channel primary
Far :example:
Near :local:
Patterns *
Expunge None
Create Both

IMAPAccount example     	  |     IMAPAccount migadu
Host localhost			  |     Host imap.migadu.com
Port 1993			  |     User sir@cmpwn.com
User user@example.com             |     Pass [...]
Pass 1234			  |     SSLType IMAPS
SSLType None			  <
IMAPStore example	          |     IMAPStore migadu
Account example			  |     Account migadu
Far :example:			  |     Far :migadu:
Patterns *			  |     Patterns INBOX Archive Sent J
Expunge None			  |     Expunge Both
Create Both			  <

There are some obvious changes here, like name, username, and password, host, and port. I’ve changed SSLType to None to match the authentication I’ve set up with emailproxy. I’ve used a wildcard for Patterns to select all remote mailbox folders, changed Expunge to None to ensure no emails are deleted on either side, and Create to Both to ensure that local mailboxes that don’t have folders are created. After running creating the top level ~/mail directory and running mbsync primary (as discussed in the original tutorial), all of my emails downloaded locally to my computer! All 500ish MB of them…

Stage 3: integrity

Turns out I didn’t need to do this! I must have not enabled the setting on my instance…

Stage 4: getting rid of spam

It turns out this is easier to do this before you download all of your emails, and they’re scattered in plaintext documents, who could have guessed? I tried to get rid of as much spam as possible, and then repeated my earlier steps. One of the things to note here is if you’re making remote changes, it might be easier to completely delete your local synced mail. If you’re doing this, you need to delete ~/mail and ~/.mbsyncrc! I got stuck here for a while…

Stage 5: switchover!

I started by testing that my emails would actually display properly, so I used FileZilla to copy one of the files from my junk folder to an existing mailbox using FTPS. It did of course work, and I saw the email in Roundcube. I upgraded my Mythic Beasts’ subscription to the plan with 25GB of hosted storage, then added my domain. This was slightly involved, to say the least.

This works easiest when Mythic Beasts can make changes for you to your DNS, the only problem is the DNS for that domain is managed by IONOS instead. IONOS wouldn’t let me transfer my domain without losing access to my WHOIS protection, so I opted to just change the nameservers and add a 3rd party registration to Mythic Beasts. I switched nameservers on IONOS to ns1.mythic-beasts.com and ns2.mythic-beasts.com. Easy enough so far.

3rd party registration requires verification, to do this, Mythic Beasts requires you either: have a verification email sent to your webmaster contact (something like webmaster or postmaster or similar at your domain), add a TXT DNS record, or some third option that I can’t remember. I didn’t read this particularly well, as I thought there was only one option, so I added an alias to webmaster for my user account on O365 (I only realised there were other options when I navigated back to the page after setting this up). I then followed through the rest of the email set-up wizard on Mythic Beasts while waiting for DNS cache timeout (there are little reminders that tell you Mythic Beasts can’t affect your changes as your domain isn’t set up with their nameservers, I disregarded these while DNS was doing DNS).

At this point, I created a new email address and mailbox from the web interface. This generated the necessary folders that I interact with by using FTPS. I initially started by just copying over all the folders to the directory named for my email address; this did not work. Roundcube doesn’t see those folders, and you instead need to copy the emails over to the Maildir directory instead. I waited until the MX records had propagated over (thanks dig), then sent and deleted an email to generate the necessary folders. I was surprised (I’m not sure why) to see everything had sent correctly! Wrapping the set-up up, I then copied the contents of the local sent directory over to its remote counterpart. Earlier I had questioned the utility of having all of these emails accessible, as you can see, I decided I did want to have access to all of these over IMAP; I have more than enough storage too!

Reflections

I completed this switch by cancelling my O365 subscription, everything should be deleted in 90 days time (it is Microsoft though, so I speculate if the contents of my emails have probably gone into training a Large Language Model). It’s not often that I embark on a project with a plan that works as well as it has here, but I’m glad I was successful here, and I’m glad to have preserved my working in the hopes it is useful for someone else in the same situation.

One small caveat of this approach was that either copying these files over to the remote server or downloading them using mbsync has modified the received time, which affected the default sort in the mail clients I tried. I opted to sort by sent time instead, for much the same effect.

Footnotes