Sending Amplitude Cohorts to Marketo — the Steps and Drawbacks of a Custom Solution

In implementations for clients, we often activate data from Amplitude Cohorts. And we’ve been asked to use Marketo Static Lists to execute that data activation. Another thing we usually do is set up a custom integration between tools in a tech stack, or evaluate whether a custom integration would be a better fit than the native one. But, no matter how good you are, custom solutions create more moving parts. This article is a case study from the field.

Key Takeaways

• Marketo is a great tool to activate the user data coming from Amplitude via an integration
• Most tools in the modern stack have native platform integrations that are seamless and easy to maintain
• Sometimes, custom integrations are necessary to get the functionality you need out of your stack
• When sending Amplitude Cohorts data to Marketo for personalization and automation, you will be limited by how the fields are synced if you only use the native connector
• Serverless functions as microservices can be used to integrate tools in your stack. AWS Lambda is our preferred option to create the middleware to handle integrations.
• The drawbacks to custom integrations involve the additional moving parts and complexity of maintaining the integration
• Custom integrations will be great for you when you know the ROI is there, and when you’re committed

Why Consider a Custom Solution for Integrating Amplitude and Marketo

As a marketing professional, you are always exploring innovative ways to optimize efforts and integrate an array of tools — a Marketo-Segment integration is a popular example. Following the marketing stack trends, most tools offer built-in integrations, enabling easy connectivity and data sharing through a few simple clicks and API keys.

These pre-configured connections provide convenience and streamline complex technology stacks, making them the preferred choice when introducing a new tool into your architecture.

However, there are instances when these native integrations fall short, necessitating a customized approach to address specific challenges. This article delves into a scenario where the existing integration between Amplitude Cohorts and Marketo’s lead database proved inadequate. We felt the need for a tailored solution.

To overcome the limitations of the native integration between Amplitude and Marketo, we developed custom middleware using AWS Lambda. The serverless functions and microservices of Lambda bridge gaps and enable seamless data exchange between platforms.

The Use Case of Activating Data from Amplitude Cohorts in Marketo Static Lists

Amplitude Cohorts are a fundamental concept used to analyze user behavior and engagement over time. A cohort is a group of users who share a common characteristic or experience within a specified period. You’ll find similar features in Adobe or in Mixpanel, common Amplitude alternatives.

Amplitude allows you to define cohorts based on various criteria, such as the date of the first app installation, user demographics, actions taken, or any other custom event or property. Amplitude makes defining user cohorts easy, so getting things set up on that side of the fence is a fairly painless process.

With user cohorts in place, the critical next step is data activation. Where is the user data going, and what is going to happen with it? This is where Marketo comes in.

By associating members of Amplitude Cohorts with leads in Marketo, you are able to leverage rich behavioral data with Marketo’s lead nurturing and communication capabilities.

You’ll create lush, personalized marketing campaigns that reach your recipients at the right point in the buyer’s journey. You’ll even be able to used advanced methodologies such as account-based lead and opportunity scoring.

The best part? There’s a native integration between Amplitude Cohorts and Marketo Static Lists. It makes it straightforward to get the two tools to start syncing and using data.

The Drawbacks of the Native Integration, or Why We Went for a Custom Solution

So what’s the catch? The problem lies in how the two systems work together. When setting up the integration between Amplitude and Marketo, a custom field needs to be created on the lead object for every cohort that is required to be synced. This isn’t necessarily a huge issue on its face, but what if you have 10 cohorts you’d like to sync?

A custom field needs to be created on the lead object for every cohort synced. What if you have 10 cohorts you’d like to sync? What about 25?

That’s going to require creating 10 new custom fields in Marketo. What about 25 cohorts? 50? You can see where I’m going with this. This type of workflow just isn’t sustainable and will inevitably end up with unusable bloat on the lead object down the road, not to mention a very anxious Marketo admin and MOPs team.

Nobody wants to have to comb through 100 boolean fields that are only used to signify list membership. This isn’t how Marketo is designed, and you or your team would be set up for trouble because it’s hard to navigate.

There’s also a difference in how the data is represented between the two systems. Amplitude is providing their resulting behavioral cohorts as a list of returned records based on predefined criteria. But once synced to Marketo, the data is specific to the lead records themselves, added as a boolean field on each record. This has a variety of implications, the biggest being how your Marketo end-users would need to structure their campaign flows and triggers.

The ideal scenario would be to have the list-based cohorts sync to Marketo Static Lists. Then, the data would be represented in the same way in both systems. So that is what we did.

The Nitty Gritty of the Custom Solution and Its Intermediary Application

We needed to develop a custom application that would serve as an intermediary between the two tools, polling and extracting from Amplitude, and then managing records in Marketo. We identified AWS Lambda as a favorable solution to help us accomplish this, with Microsoft Azure as a viable alternative. We created custom middleware as a serverless function and had it sit between Amplitude and Marketo. Let’s now geek out about the technical details.

Triggered on a regularly-recurring batched basis, the serverless function queries Amplitude’s behavioral cohorts API for all of the active cohorts in the instance and then locates specific cohorts based on a naming convention we defined. By designing it this way, we were able to allow end-users in Amplitude to decide which cohorts would be synced to Marketo and which would not, as opposed to every cohort syncing.

Additionally, the function is using environment variables (parameters) in AWS to delineate which Amplitude project and Marketo instance to use, so users have access to additional configuration options here as well.

AMPLITUDE_USERNAME=''
AMPLITUDE_PASSWORD=''
AMPLITUDE_EMAIL_FIELD='gp:email'

MARKETO_CLIENT_ID=''
MARKETO_CLIENT_SECRET=''
MARKETO_IDENTITY_URL='https://[ACCOUNTSTRING].mktorest.com/identity'
MARKETO_REST_URL='https://[ACCOUNTSTRING].mktorest.com/rest'
MARKETO_BULK_URL=''
MARKETO_COHORTS_FOLDER_ID=''

This is the complicated part: once the process runs and has a list of all the cohorts that need to be synced, it sends those lists back to the Behavioral Cohorts API to export their members. That API call returns a request_id, which needs to be sent back to the Cohorts API to download the cohort members from Amplitude’s S3 bucket.

Developing this part of the application, in particular, was a challenge. The export needs some time to process before it can be downloaded, depending on how many members are in the cohort. If the download request is sent before the export is finished processing, an error will be returned, and the download will be skipped.

If you continuously retry the download request until it’s eventually done processing, you will risk eating into your API call allocation, which is fairly low. At the time of writing, the monthly API calls to the Behavioral Cohorts API are limited to 500 per organization.

So even if you have different projects that use the API, they all count against that same limit. Doing all of this asynchronously took a lot of trial and error to find the right sweet spot to get everything to communicate as expected.

Once the cohorts are downloaded from Amplitude’s S3 storage, the application then makes an API request to Marketo for all the static lists in that particular instance with the same naming convention defined for the Amplitude Cohorts. That way, we’re able to tell whether or not a cohort has already been synced to Marketo. If it doesn’t already exist, the application creates a list. Otherwise, this step is skipped.

From here, the application uses a combination of Marketo’s Bulk Lead Import API and Assets API for Static List membership to add or remove records from them based on their presence in the cohorts at that particular time. The nuance here was ensuring we weren’t only adding leads to lists if they existed in cohorts but also removing them if they weren’t.

Ensuring you’re not only adding leads to lists if they exist in cohorts but also removing them if they don’t.

This complex list-membership logic checking was another aspect of the application that required a lot of trial and error to get it right.

Once we overcame the challenges above, we had a custom solution that worked. Would we do it that way again?

The Drawbacks of the Custom Amplitude-Marketo Integration

Creating a custom integration between tools does not come without its downsides. Consider the following:

1. Complexity and maintenance: Developing a custom solution requires significant time, effort, and expertise. It involves designing, implementing, and testing the solution from scratch. Additionally, as software and systems evolve, the custom solution may require ongoing maintenance and updates, which can be resource-intensive.

2. Compatibility issues: A custom solution may face compatibility challenges when integrating with existing systems or software. It may require extensive modifications or adaptations to ensure seamless interoperability. This can lead to delays, additional costs, and potential disruptions to business operations.

3. Limited functionality: Custom solutions often focus on specific requirements and may lack the comprehensive features and functionalities provided by native integrations. Native integrations are built by the creators of the software or platform and are designed to work seamlessly within the existing ecosystem, offering a more comprehensive range of capabilities.

4. Time and cost implications: Developing a custom solution can be time-consuming and costly. It involves gathering requirements, designing the solution, coding, testing, and deployment. On the other hand, native integrations are typically pre-built and readily available, saving time and reducing development costs.

5. Future scalability and upgrades: Custom solutions may face challenges when it comes to scalability and future upgrades. As your business grows or evolves, the custom solution may struggle to accommodate new requirements or changes in technology. In contrast, native integrations are often designed to evolve with the platform, providing regular updates and enhancements.
6. Support and documentation: Native integrations usually come with comprehensive documentation and dedicated support channels provided by the platform or software provider. In contrast, with a custom solution, support may be limited to the development team or external consultants involved in building the solution. This can lead to difficulties in troubleshooting issues and obtaining timely support when needed.

When Should You Opt for a Custom Integration?

Custom integrations are not just something you can forget about because they take more to maintain. Your organization will benefit from them big time when all of the below are present:

1. The data activation, in other words, the action you’ll be able to take thanks to your custom integration, will have an ROI higher than the cost of building and maintaining it
2. You’re going to document the custom integration thoroughly
3. You’re willing to keep maintaining it long-term
4. You’ll keep your eyes open to improvements in native integrations that could render your custom integration obsolete

What custom solutions have you developed? Let us know in the comments.