'Approach for syncing data from different CRM/MAP tools for different clients

We are bulding a highly efficient marketing automation tool.

What we want is to periodically sync data/modified data from our customer's CRM/MAP tools for salesforce, marketo, hubspot, etc.

While we have the REST APIs and all to do that, the biggest challenge we are facing is on architectural side. We are completely on AWS.

We have used sqs-lambda approach - but maintaing parallel syncing is a challenge. Also, it is difficult to modify changes with this approach as and when the number of customers and their integrations with CRM/MAP tools increase.

We are trying to use Airflow (MWAA) with FARGATE now - But we are facing some issues there as well.

Note:

  • Some syncs are being processed and its time taking (Hence, Lambda is not a feasible approach)

Please suggest some good approach to do this efficiently on a daily basis.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source