MS Office 365 Exchange OAuth2 IMAP Authentication with Flowable - Part 1 Azure Setup

Over the last months, Microsoft has been gradually deprecating and disabling basic authentication for Microsoft Exchange Online protocols like POP and IMAP.

The last opportunity to switch to the new, modern, OAuth 2.0-based authentication is December 2022, very soon. It sounds like a simple change, but it turned out to be inconvenient from a configuration point of view in practice at some points.

Fear not - we went through this, got it to work with Flowable, and share our knowledge in our typical open source way, to save you a couple of hours. Which you hopefully can use in a better way.

This blog post is helpful for everyone dealing with programmatic e-mail access using the Java javax.mail APIs or who is using Flowable's Inbound E-Mail channel to listen for e-mails over IMAP for Microsoft Office 365 Exchange Online mailboxes. We show programmatic access, Spring configurations, and Flowable configurations to access e-mails over IMAP.

NOTE: This guide covers the Client Credentials Grant flow, usually used for machine-to-machine communication without user interaction.

This post is part of the series. In the second part, we show how to configure Flowable and Spring Boot, and how to use plain Spring and javax.mail to access a mailbox over IMAP.

STEP 1 - Register / Setup the App in Azure

The Azure App manages the identity and access interface to our Mailboxes, which we want to access from our Java or Flowable application using modern authentication with OAuth2 Client Credentials flow.

To access e-mails over IMAP the app must be configured correctly. In short, the following pre-conditions must be fulfilled: (Find the step-by-step guide at the end of this article)

• Under Permissions, there must be an IMAP.AccessAsApp permission with a green checkmark:

  

• Under Certificates & secrets, there must be a Client secret or Certificate configured. You need the secret or the certificate at hand.

If you are an Azure rookie and cannot follow the pre-conditions above, head down to the bottom of this article for a step-by-step guide on how to create and setup the App registration.

Why?

This setup allows you to obtain an access token using the OAuth2 client credentials grant flow for your registered App. This access token is used as the password for the IMAP connection later on. OAuth2 is a wide-spread, quite complex, and somewhat overwhelming standard, with a lot of stuff to understand in its entirety. However, for this use case, you don't have to understand the details. We basically just need the clientId, clientSecret or certificate, and the Azure tenant name or tenantId.

STEP 2 - Add-MailboxPermissions using PowerShell command

This was the most complicated part to get right, because I missed a detail in the Microsoft documentation, which I am trying to emphasize here. An Azure admin is required to execute those steps. Time to seek out the Azure administrator of your company, in case you are not that guy.

Next, we are going to create a servicePrincipal and execute Add-MailboxPermission to permit access to the mailbox … which ... is only possible by executing some PowerShell magic. For this, we require a special Object ID located at very different places. Both places contain an Object ID, but they are different. Confusing? Seems like many did not get this right in the first place. We try to make it clearer:

We are going to need the Application (client) ID from the application overview page, Please IGNORE the Object ID there and get the ENTERPRISE_OBJECT_ID from the location shown below:

The ENTERPRISE Object ID can be found under Enterprise applications -> All applications -> Type application name in the search box:

Ignore the crossed-out Object ID in red from the application overview, but get the Enterprise Object Id, from the Enterprise applications entry point in Azure instead.

Here we go with the PowerShell commands:

# Registering an Azure AD application's service principal in Exchange: 
New-ServicePrincipal -AppId "<APPLICATION_ID>" -ServiceId "<ENTERPRISE_OBJECT_ID>"

# Set the DisplayName of the newly created ServicePrincipal 
Set-ServicePrincipal -Identity "<ENTERPRISE_OBJECT_ID>" -DisplayName "<APP_DISPLAY_NAME>"

# Give the application's service principal access to one mailbox: 
# Add-MailboxPermission -Identity "<EMAIL_ADDRESS>" -User "<ENTERPRISE_OBJECT_ID>" -AccessRights FullAccess 
Add-MailboxPermission-Identity "<EMAIL_ADDRESS>" -User "<ENTERPRISE_OBJECT_ID>" -AccessRightsFullAccess

You might need to install some PowerShell modules to execute those commands first. The Microsoft description can be found here:

https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#register-service-principals-in-exchange

STEP 3 - Testing time!

I created a small JBang! command line testing tool to quickly test out the configuration. The source code is available by clicking the link below. Using the -v (verbose) mode, it prints valuable debug information, in case it does not work.

DISCLAIMER: This tool has not been tested thoroughly and should be just seen as a goodie or code inspiration, to save you some time. It has served me well but is not product-quality. Please treat it as an open source snippet, with a "sharing is caring" mindset.

Download and install JBang and run:

$ jbang https://raw.githubusercontent.com/flowable/flowable-examples/master/blog/2022-11-ms-exchange-online-modern-auth/src/main/java/org/flowable/example/blog/MsExchangeOnlineModernAuthTestCli.java <tenantIdOrDomain> -c=<clientId> -cs=<clientSecret> -e <emailAddress>

Or follow the link above to get the source code, check it out and run the class available in src/test/java directory.

At your own risk, you can run it directly using the command below, without installation of JBang (for bash-based systems):

$ curl -Ls https://sh.jbang.dev | bash -s - https://raw.githubusercontent.com/flowable/flowable-examples/master/blog/2022-11-ms-exchange-online-modern-auth/src/main/java/org/flowable/example/blog/MsExchangeOnlineModernAuthTestCli.java <tenantIdOrDomain> -c=<clientId> -cs=<clientSecret> -e <emailAddress>

Testing tool example usage and output

$ jbang https://raw.githubusercontent.com/flowable/flowable-examples/master/blog/2022-11-ms-exchange-online-modern-auth/src/main/java/org/flowable/example/blog/MsExchangeOnlineModernAuthTestCli.java flowable.com -c=84464XXXXXXX4f-b09XXX86 -cs=RcsXXXXXXXNjdey -e mailbox-testing@test.flowable.com

Trying to fetch access token ...
[OK] SUCCESS! Got OAuth2 access token. Token expires: Tue Nov 22 18:31:40 CET 2022
Print token? Use -v option.
Microsoft gave us an access token. That is halfway. Now trying to access mailbox 'bot-mailbox-testing@bots.flowable.com' for folder 'Inbox' over IMAP.
Trying to access IMAP mailbox with properties
mail.imaps.auth=true
mail.imaps.port=993
mail.imaps.auth.plain.disable=true
mail.store.protocol=imaps
mail.imaps.user=bot-mailbox-testing@bots.flowable.com
mail.imaps.auth.mechanisms=XOAUTH2
mail.imaps.ssl.enable=true
mail.imaps.auth.xoauth2.disable=false
mail.imaps.host=outlook.office365.com
mail.imaps.starttls.enable=true
[OK] Mail Store connected.
Now opening Inbox
[OK] SUCCESS! IMAP Folder Inbox opened.
[OK] Nr. of Messages in Mailbox: 4
------------ Listing Mails ----------------
[OK]  > Mail Received: Arthur Hupka-Merle <arthur.hupka-merle@flowable.com> -[2022-11-15 17:49:03] From: Test Mail
[OK]  > Mail Received: Arthur Hupka-Merle <arthur.hupka-merle@flowable.com> -[2022-11-16 15:28:34] From: Test 2 Process
[OK]  > Mail Received: Arthur Hupka-Merle <arthur.hupka-merle@flowable.com> -[2022-11-16 15:49:35] From: How are you?
[OK]  > Mail Received: Arthur Hupka-Merle <arthur.hupka-merle@flowable.com> -[2022-11-17 16:59:29] From: My Mail
------------ Done Listing ----------------
[OK] Congrats! Your Mailbox mailbox-testing@test.flowable.com is ready to go with OAuth2 authentication.
Let the flow be with you. Brought to you by flowable.com

The output above shows example usage and output for a working connection.

Troubleshoot

Does something fail? Authentication errors? Add the '-v' (verbose) flag to the testing tool, to get detailed output.

A1 NO AUTHENTICATE failed

If you get the error below it is likely, that something went wrong during STEP 2 of this guide and the servicePrincipal has insufficient rights. One reason could be, that the admin did not use the correct Enterprise Object ID

A1 NO AUTHENTICATE failed
javax.mail.AuthenticationFailedException: AUTHENTICATE failed

See discussion here: https://stackoverflow.com/questions/73463357/cannot-authenticate-to-imap-on-office365-using-javamail

Conclusion

This part lays the foundation and shows the required Azure app configurations to access a mailbox using modern authentication for Microsoft Office 365 Exchange Online, which will be the only one supported very soon. It shows pitfalls during the configuration of the OAuth2 Client Credentials flow in Azure and Exchange Online. Hopefully, it is going to help some of you, which have been entrusted to migrate to this authentication method, to avoid those pitfalls. Furthermore, the testing tool provides an easy and fast way to test if the configuration works and allows access to the mailbox.

Alright, how to configure Flowable?

That's it for the test setup and the first part of this Blog Post. Head over to part two, which showcases and discusses how to use Flowable's Inbound E-Mail Channel to access an Office 365 Mailbox using this authentication method and also shows how to configure Spring Boot and use plain javax.mail API.

Appendix: Register the App in Azure for IMAP access Step by Step

This part can be skipped, if an App is already registered and set up and has been configured already by an Azure admin.

NOTE: This is the guide to setting up Client Credentials Grant flow only, which is usually used for machine-to-machine communication, without user interaction.

1. Create the App

2. Add required IMAP.AccessAsApp permission

Under API permissions add the IMAP.AccessAsApp permission. You can find it under APIs my organization uses > Office 365 Exchange Online.

Then select Application permissions and look for IMAP:

NOTE: This permission requires Admin consent. If you are not the admin reach out to your admin and request consent to it. This is how it should look like:

3. Create Client Secret or Client Certificate

To connect to this App registration over OAuth2, a Client Secret or a Client Certificate (public key) is required.

For simplicity reasons, we use a client secret here.

NOTE: Take care of the client secret expiration date! Or define a custom period. When the key expires, your client application won’t be able to connect anymore.

NOTE: Copy and store the Client Secret right after it has been created to a secure place e.g., secure key storage. You won’t be able to see it again.

Go to part 2