Email may not be as cool as other communication platforms but working with it can still be fun. I was recently tasked with implementing messaging in a mobile app. The only catch was that the actual communication needed to be over email. We wanted app users to be able to communicate with a support team just like you would send a text message. Support team members needed to receive these messages via email, and also needed to be able to respond to the originating user. To the end user, everything needed to look and function just like any other modern messaging app.
In this article, we will take a look at how to implement a service similar to the one described above using Java and a handful of Amazonโs web services. You will need a valid AWS account, a domain name, and access to your favorite Java IDE.
The Infrastructure
Before we write any code, weโre going to set up the required AWS services for routing and consuming email. Weโre going to use SES for sending and consuming emails and SNS+SQS for routing incoming messages.
It all starts here with SES. Start by logging into your AWS account and navigating to the SES console.
Before we begin, youโre going to need a verified domain name you can send emails from.
This will be the domain app users will be sending email messages from and support members will be replying to. Verifying a domain with SES is a straightforward process, and more info can be found here.
If this is the first time you are using SES, or you have not requested a sending limit, your account will be sandboxed. This means that you will not be able to send email to addresses that arenโt verified with AWS. This may cause an error later in the tutorial, when we send an email to our fictional help desk. To avoid this, you can verify whatever email address you plan on using as your help desk in the SES console in the Email Addresses tab.
Once you have a verified domain, we can create a rule set. Navigate to the Rule Sets tab in the SES console and create a new Receipt Rule.
The first step when creating a receipt rule will be defining a recipient.
Recipients filters will allow you to define what emails SES will consume, and how to process each incoming message. The recipient we define here needs to match the domain and address pattern app user messages are emailed from. The simplest case here would be to add a recipient for the domain we previously verified, in our case example.com. This will configure SES to apply our rule to all emails sent to example.com. (e.g. foo@example.com, bar@example.com).
To create a rule for our entire domain, we would add a recipient for example.com.
Itโs also possible to match address patterns. This is useful if you want to route incoming messages to different SQS queues.
Say that we have queue A and queue B. We could add two recipients: a@example.com and b@example.com. If we want to insert a message into queue A, we would email a+foo@example.com. The a part of this will match our a@example.com recipient. Everything between the + and @ is arbitrary user data, it will not affect SESโs address matching. To insert into queue B, simply replace a with b.
After you define your recipients, the next step is to configure the action SES will perform after consuming a new email. We eventually want these to end up in SQS, however it is currently not possible to go directly from SES to SQS. To bridge the gap, we need to use SNS. Select the SNS action and create a new topic. We will eventually configure this topic to insert messages into SQS.
Select create SNS topic and give it a name.
After the topic is created, we need to select a message encoding. Iโm going to use Base64 in order to preserve special characters. The encoding you choose will affect how messages are decoded when we consume them in our service.
Once the rule is set, we just need to name it.
The next step will be configuring SQS and SNS, for that we need to head over to the SQS console and create a new queue.
To keep things simple, Iโm using the same name as our SNS topic.
After we define our queue, weโre going to need to adjust its access policy. We only want to grant our SNS topic permission to insert. We can achieve this by adding a condition that matches our SNS topic arn.
The value field should be populated with the ARN for the SNS topic SES is notifying.
After SQS is set up, itโs time for one more trip back to the SNS console to configure your topic to insert notifications into your shiny new SQS queue.
In the SNS console, select the topic SES is notifying. From there, create a new subscription. The subscription protocol should be Amazon SQS, and the destination should be the ARN of the SQS queue you just generated.
After all that, the AWS side of the equation should be all set up. We can test our work by emailing ourselves. Send an email to the domain configured with SES, then head to the SQS console and select your queue. You should be able to see the payload containing your email.
Java Service to Deal with Emails
Now on to the fun part! In this section, weโre going to create a simple microservice capable of sending messages and processing incoming emails. The first step will be defining an API that will email our support desk on behalf of a user.
A quick note. Weโre going to focus on the business logic components of this service, and wonโt be defining REST endpoints or a persistence layer.
To build a Spring service, weโre going to use Spring Boot and Maven. We can use Spring Initializer to generate a project for us, start.spring.io.
To start, our pom.xml should look something like this:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.toptal.tutorials</groupId>
<artifactId>email-processor</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>email-processor</name>
<description>A simple "micro-service" for emailing support on behalf of a user and processing replies</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.3.5.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Emailing Support on Behalf of a User
First, letโs define a bean for emailing our support desk on behalf of a user. The job of this bean will be to process an incoming message from a user ID, and email that message to our pre-defined support desk email address.
Letโs start by defining an interface.
public interface SupportBean {
/**
* Send a message to the application support desk on behalf of a user
* @param fromUserId The ID of the originating user
* @param message The message to send
*/
void messageSupport(long fromUserId, String message);
}
And an empty implementation:
@Component
public class SupportBeanSesImpl implements SupportBean {
/**
* Email address for our application help-desk
* This is the destination address user support emails will be sent to
*/
private static final String SUPPORT_EMAIL_ADDRESS = "support@example.com";
@Override
public void messageSupport(long fromUserId, String message) {
//todo: send an email to our support address
}
}
Letโs also add the AWS SDK to our pom, weโre going to use the SES client to send our emails:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.5</version>
</dependency>
The first thing we need to do is generate an email address to send our userโs message from. The address we generate will play a critical role on the consuming side of our service. It needs to contain enough information to route the help deskโs reply back to the originating user.
To achieve this, weโre going to include the originating user ID in our generated email address. To keep things clean, weโre going to create an object containing the user ID and use the Base64 encoded JSON string of it as the email address.
Letโs create a new bean responsible for turning a user ID into an email address.
public interface UserEmailBean {
/**
* Returns a unique per user email address
* @param userID Input user ID
* @return An email address unique for the input userID
*/
String emailAddressForUserID(long userID);
}
Letโs start our implementation by adding the required consents and a simple inner class that will help us serialize our JSON.
@Component
public class UserEmailBeanJSONImpl implements UserEmailBean {
/**
* The TLD for all our generated email addresses
*/
private static final String EMAIL_DOMAIN = "example.com";
/**
* com.fasterxml.jackson.databind.ObjectMapper used to create a JSON object including our user ID
*/
private final ObjectMapper objectMapper = new ObjectMapper();
@Override
public String emailAddressForUserID(long userID) {
//todo: create the email address
return null;
}
/**
* Simple helper class we will serialize.
* The JSON representation of this class will become our user email address
*/
private static class UserDetails{
private Long userID;
public Long getUserID() {
return userID;
}
public void setUserID(Long userID) {
this.userID = userID;
}
}
}
Generating our email address is straightforward, all we need to do is create a UserDetails object and Base64 encode the JSON representation. The finished version of our createAddressForUserID method should look something like this:
@Override
public String emailAddressForUserID(long userID) {
UserDetails userDetails = new UserDetails();
userDetails.setUserID(userID);
//create a JSON representation.
String jsonString = objectMapper.writeValueAsString(userDetails);
//Base64 encode it
String base64String = Base64.getEncoder().encodeToString(jsonString.getBytes());
//create an email address out of it
String emailAddress = base64String + "@" + EMAIL_DOMAIN;
return emailAddress;
}
Now we can head back to SupportBeanSesImpl and update it to use the new email bean we just created.
private final UserEmailBean userEmailBean;
@Autowired
public SupportBeanSesImpl(UserEmailBean userEmailBean) {
this.userEmailBean = userEmailBean;
}
@Override
public void messageSupport(long fromUserId, String message) throws JsonProcessingException {
//user specific email
String fromEmail = userEmailBean.emailAddressForUserID(fromUserId);
}
To send emails, weโre going to use the AWS SES client included with the AWS SDK.
/**
* SES client
*/
private final AmazonSimpleEmailService amazonSimpleEmailService = new AmazonSimpleEmailServiceClient(
new DefaultAWSCredentialsProviderChain() //see http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
);
Weโre utilizing the DefaultAWSCredentialsProviderChain to manage credentials for us, this class will search for AWS credentials as defined here.
Weโre going to an AWS access key provisioned with access to SES and eventually SQS. For more info check out the documentation from Amazon.
The next step will be updating our messageSupport method to email support using the AWS SDK. The SES SDK makes this a straightforward process. The finished method should look something like this:
@Override
public void messageSupport(long fromUserId, String message) throws JsonProcessingException {
//User specific email
String fromEmail = userEmailBean.emailAddressForUserID(fromUserId);
//create the email
Message supportMessage = new Message(
new Content("New support request from userID " + fromUserId), //Email subject
new Body().withText(new Content(message)) //Email body, this contains the userโs message
);
//create the send request
SendEmailRequest supportEmailRequest = new SendEmailRequest(
fromEmail, //From address, our user's generated email
new Destination(Collections.singletonList(SUPPORT_EMAIL_ADDRESS)), //to address, our support email address
supportMessage //Email body defined above
);
//Send it off
amazonSimpleEmailService.sendEmail(supportEmailRequest);
}
To try it out, create a test class and inject the SupportBean. Make sure SUPPORT_EMAIL_ADDRESS defined in SupportBeanSesImpl points to an email address you own. If your SES account is sandboxed, this address also needs to be verified. Email addresses can be verified in the SES console under Email Addresses section.
@Test
public void emailSupport() throws JsonProcessingException {
supportBean.messageSupport(1, "Hello World!");
}
After running this, you should see a message show up in your inbox. Better yet, reply to the message and check the SQS queue we set up earlier. You should see a payload containing your reply.
Consuming Replies from SQS
The last step will be to read in emails from SQS, parse out the email message, and figure out what user ID the reply should be forwarded belongs to.
To listen for new SQS messages, weโre going to use the Spring Cloud AWS messaging SDK. This will allow us to configure a SQS message listener via annotations, and thus avoid quite a bit of boilerplate code.
First, the required dependencies.
Add the Spring Cloud messaging dependency:
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-aws-messaging</artifactId>
</dependency>
And add Spring Cloud AWS to your pom dependency management:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-aws</artifactId>
<version>1.1.0.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Currently, Spring Cloud AWS doesnโt support annotation driven configuration, so weโre going to have to define an XML bean. Luckily we donโt need much configuration at all, so our bean definition will be pretty light. The main point of this file will be to enable annotation driven queue listeners, this will allow us to annotate a method as an SqsListener.
Create a new XML file named aws-config.xml in your resources folder. Our definition should look something like this:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:aws-context="http://www.springframework.org/schema/cloud/aws/context"
xmlns:aws-messaging="http://www.springframework.org/schema/cloud/aws/messaging"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/cloud/aws/context
http://www.springframework.org/schema/cloud/aws/context/spring-cloud-aws-context.xsd
http://www.springframework.org/schema/cloud/aws/messaging
http://www.springframework.org/schema/cloud/aws/messaging/spring-cloud-aws-messaging.xsd">
<!--enable annotation driven queue listeners -->
<aws-messaging:annotation-driven-queue-listener />
<!--define our region, this lets us reference queues by name instead of by URL. -->
<aws-context:context-region region="us-east-1" />
</beans>
The important part of this file is <aws-messaging:annotation-driven-queue-listener />
. We are also defining a default region. This is not necessary, but doing so will allow us to reference our SQS queue by name instead of URL. We are not defining any AWS credentials, by omitting them Spring will default to DefaultAWSCredentialsProviderChain, the same provider we used earlier in our SES bean. More info can be found in the Spring Cloud AWS docs.
To use this XML config in our Spring Boot app, we need to explicitly import it. Head over to your @SpringBootApplication class and import it.
@SpringBootApplication
@ImportResource("classpath:aws-config.xml") //Explicit import for our AWS XML bean definition
public class EmailProcessorApplication {
public static void main(String[] args) {
SpringApplication.run(EmailProcessorApplication.class, args);
}
}
Now letโs define a bean that will handle incoming SQS messages. Spring Cloud AWS lets us accomplish this with a single annotation!
/**
* Bean reasonable for polling SQS and processing new emails
*/
@Component
public class EmailSqsListener {
@SuppressWarnings("unused") //IntelliJ isn't quite smart enough to recognize methods marked with @SqsListener yet
@SqsListener(value = "com-example-ses", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS) //Mark this method as a SQS listener
//Since we already set up our region we can use the logical queue name here
//Spring will automatically delete messages if this method executes successfully
public void consumeSqsMessage(@Headers Map<String, String> headers, //Map of headers returned when requesting a message from SQS
//This map will include things like the relieved time, count and message ID
@NotificationMessage String rawJsonMessage //JSON string representation of our payload
//Spring Cloud AWS supports marshaling here as well
//For the sake of simplicity we will work with incoming messages as a JSON object
) throws Exception{
//com.amazonaws.util.json.JSONObject included with the AWS SDK
JSONObject jsonSqsMessage = new JSONObject(rawJsonMessage);
}
}
The magic here lies with the @SqsListener annotation. With this, Spring will set up an Executor and start polling SQS for us. Every time a new message is found, our annotated method will be invoked with the message contents. Optionally, Spring Cloud can be configured to marshall incoming messages, giving you the ability to work with strong typed objects inside your queue listener. Additionally, you have the ability to inject a single header or a map of all headers returned from the underlying AWS call.
Weโre able to use the logical queue name here since we previously defined the region in aws-config.xml, if we wanted to omit that we would be able to replace the value with our fully qualified SQS URL. Weโre also defining a deletion policy, this will configure Spring to delete the incoming message from SQS if its condition is met. There are multiple policies defined in SqsMessageDeletionPolicy, weโre configuring Spring to delete our message if our consumeSqsMessage method executes successfully.
Weโre also injecting the returned SQS headers into our method using @Headers, and the injected map will contain metadata related to the queue and payload received. The message body is injected using @NotificationMessage. Spring supports marshalling utilizing Jackson, or via a custom message body converter. For the sake of convenience, weโre just going to inject the raw JSON string and work with it using the JSONObject class included with the AWS SDK.
The payload retrieved from SQS will contain a lot of data. Take a look at the JSONObject to familiarize yourself with the payload returned. Our payload contains data from every AWS service it was passed through, SES, SNS, and finally SQS. For the sake of this tutorial, we really only care about two things: the list of email addresses this was sent to and the email body. Letโs start by parsing out the emails.
//Pull out the array containing all email addresses this was sent to
JSONArray emailAddressArray = jsonSqsMessage.getJSONObject("mail").getJSONArray("destination");
for(int i = 0 ; i < emailAddressArray.length() ; i++){
String emailAddress = emailAddressArray.getString(i);
}
Since in the real world, our helpdesk may include more than just the original sender in his or her reply, weโre going to want to verify the address before we parse out the user ID. This will give our support desk both the ability to message multiple users at the same time as well as the ability to include non app users .
Letโs head back over to our UserEmailBean interface and add another method.
/**
* Returns true if the input email address matches our template
* @param emailAddress Email to check
* @return true if it matches
*/
boolean emailMatchesUserFormat(String emailAddress);
In UserEmailBeanJSONImpl, to implement this method weโre going to want to do two things. First, check if the address ends with our EMAIL_DOMAIN, then check if we can marshall it.
@Override
public boolean emailMatchesUserFormat(String emailAddress) {
//not our address, return right away
if(!emailAddress.endsWith("@" + EMAIL_DOMAIN)){
return false;
}
//We just care about the email part, not the domain part
String emailPart = splitEmail(emailAddress);
try {
//Attempt to decode our email
UserDetails userDetails = objectMapper.readValue(Base64.getDecoder().decode(emailPart), UserDetails.class);
//We assume this email matches if the address is successfully decoded and marshaled
return userDetails != null && userDetails.getUserID() != null;
} catch (IllegalArgumentException | IOException e) {
//The Base64 decoder will throw an IllegalArgumentException it the input string is not Base64 formatted
//Jackson will throw an IOException if it can't read the string into the UserDetails class
return false;
}
}
/**
* Splits an email address on @
* Returns everything before the @
* @param emailAddress Address to split
* @return all parts before @. If no @ is found, the entire address will be returned
*/
private static String splitEmail(String emailAddress){
if(!emailAddress.contains("@")){
return emailAddress;
}
return emailAddress.substring(0, emailAddress.indexOf("@"));
}
We defined two new methods, emailMatchesUserFormat which we just added to our interface, and a simple utility method for splitting an email address on the @. Our emailMatchesUserFormat implementation works by attempting to Base64 decode and marshall the address part back into our UserDetails helper class. If this succeeds, we then check to make sure the required userID is populated. If all this works out, we can safely assume a match.
Head back to our EmailSqsListener and inject the freshly updated UserEmailBean.
private final UserEmailBean userEmailBean;
@Autowired
public EmailSqsListener(UserEmailBean userEmailBean) {
this.userEmailBean = userEmailBean;
}
Now weโre going to update the consumeSqsMethod. First letโs parse out the email body:
//Pull our content, remember the content will be Base64 encoded as per our SES settings
String encodedContent = jsonSqsMessage.getString("content");
//Create a new String after decoding our body
String decodedBody = new String(
Base64.getDecoder().decode(encodedContent.getBytes())
);
for(int i = 0 ; i < emailAddressArray.length() ; i++){
String emailAddress = emailAddressArray.getString(i);
}
Now letโs create a new method that will process the email address and email body.
private void processEmail(String emailAddress, String emailBody){
}
And finally, update the email loop to invoke this method if it finds a match.
//Loop over all sent to addresses
for(int i = 0 ; i < emailAddressArray.length() ; i++){
String emailAddress = emailAddressArray.getString(i);
//If we find a match, process the email and method
if(userEmailBean.emailMatchesUserFormat(emailAddress)){
processEmail(emailAddress, decodedBody);
}
}
Before we implement processEmail, we need to add one more method to our UserEmailBean. We need a method for returning the userID from an email. Head back over to the UserEmailBean interface to add its last method.
/**
* Returns the userID from a formatted email address.
* Returns null if no userID is found.
* @param emailAddress Formatted email address, this address should be verified using {@link #emailMatchesUserFormat(String)}
* @return The originating userID if found, null if not
*/
Long userIDFromEmail(String emailAddress);
The goal of this method will be to return the userID from a formatted address. The implementation will be similar to our verification method. Letโs head over to UserEmailBeanJSONImpl and fill in this method.
@Override
public Long userIDFromEmail(String emailAddress) {
String emailPart = splitEmail(emailAddress);
try {
//Attempt to decode our email
UserDetails userDetails = objectMapper.readValue(Base64.getDecoder().decode(emailPart), UserDetails.class);
if(userDetails == null || userDetails.getUserID() == null){
//We couldn't find a userID
return null;
}
//ID found, return it
return userDetails.getUserID();
} catch (IllegalArgumentException | IOException e) {
//The Base64 decoder will throw an IllegalArgumentException it the input string is not Base64 formatted
//Jackson will throw an IOException if it can't read the string into the UserDetails class
//Return null since we didn't find a userID
return null;
}
}
Now head back over to our EmailSqsListener and update processEmail to use this new method.
private void processEmail(String emailAddress, String emailBody){
//Parse out the email address
Long userID = userEmailBean.userIDFromEmail(emailAddress);
if(userID == null){
//Whoops, we couldn't find a userID. Abort!
return;
}
}
Great! Now we have almost everything we need. The last thing we need to do is parse out the reply from the raw message.
Parsing out replies from emails is actually a fairly complicated task. Email message formats are not standardized, and the variations between different email clients can be huge. The raw response is also going to include much more than the reply and a signature. The original message will most likely be included as well. Smart people over at Mailgun put together a great blog post explaining some of the challenges. They also open sourced their machine-learning based approach to parsing emails, check it out here.
The Mailgun library is written in Python, so for our tutorial weโre going to use a simpler Java based solution. GitHub user edlio put together an MIT licensed email parser in Java based on one of GitHubโs libraries. Weโre going to use this great library.
First letโs update our pom, weโre going to use https://jitpack.io to pull in EmailReplyParser.
<repositories>
<repository>
<id>jitpack.io</id>
<url>https://jitpack.io</url>
</repository>
</repositories>
Now add the GitHub dependency.
<dependency>
<groupId>com.github.edlio</groupId>
<artifactId>EmailReplyParser</artifactId>
<version>v1.0</version>
</dependency>
Weโre also going to use Apache commons email. Weโre going to need to parse the raw email into a javax.mail MimeMessage before passing it off to the EmailReplyParser. Add the commons dependency.
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-email</artifactId>
<version>1.4</version>
</dependency>
Now we can head back over to our EmailSqsListener and finish up processEmail. At this point, we have the originating userID and the raw email body. The only thing left to do is parse out the reply.
To accomplish this, weโre going to use a combination of javax.mail and edlioโs EmailReplyParser.
private void processEmail(String emailAddress, String emailBody) throws Exception {
//Parse out the email address
Long userID = userEmailBean.userIDFromEmail(emailAddress);
if(userID == null){
//Whoops, we couldn't find a userID. Abort!
return;
}
//Default javax.mail session
Session session = Session.getDefaultInstance(new Properties());
//Create a new mimeMessage out of the raw email body
MimeMessage mimeMessage = MimeMessageUtils.createMimeMessage(
session,
emailBody
);
MimeMessageParser mimeMessageParser = new MimeMessageParser(mimeMessage);
//Parse the message
mimeMessageParser.parse();
//Parse out the reply for our message
String replyText = EmailReplyParser.parseReply(mimeMessageParser.getPlainContent());
//Now we're done!
//We have both the userID and the response!
System.out.println("Processed reply for userID: " + userID + ". Reply: " + replyText);
}
Wrap Up
And thatโs it! We now have everything we need to deliver a response to the originating user!
In this article, we saw how Amazon Web Services can be used to orchestrate complex pipelines. Although in this article, the pipeline was designed around emails; these same tools can be leveraged to design even more complex systems, where you donโt have to worry about maintaining the infrastructure and can focus on the fun aspects of software engineering instead.
This article was written byย Francis Altomare, a Toptalย Java developer.
Discover more from TechBooky
Subscribe to get the latest posts sent to your email.