Distributed Systems - Events Consumption | Sync Problem

Question

In one of the interviews, I was asked to build a payment ecosystem, which had 2 services - Activity Service and Credit Service. Activity Service is responsible for providing user activity information to the customer. Credit Service is responsible for providing credit related data.

Both these services listen to the same PAYMENT event from a kafka topic. Function of Activity service is listen to the PAYMENT event and make an API call to credit service to fetch credit data and store this aggregated view of PAYMENT event plus credit data into the Activity Database.

Whereas Credit Service also listens to the PAYMENT event and it updates its database based on this event.

One of the problem that is currently faced by the system is that, by the time Activities Service make an API call to Credit Service on consumption of PAYMENT event, the information provided by the Credit Service is not the updated, reason being Credit Service hasn't updated it's database based on the same PAYMENT event.

This looks like classic distributed systems problem. Any insights into how to solve this problem will be highly helpful.

Answer 1

Since it's useful for at least one other service to have both a payment event and the updates from the credit service arising from that event, one reasonable solution might be to have the credit service publish an event which is just the original payment event enriched with the updates the credit service made to its model (or possibly, if consumers are expected to query the credit service, just the fact that it has updated the model).

Then the activity service can subscribe to those enriched events and either:

• know that the credit service has processed the corresponding payment event so therefore the credit service's data model has seen the payment event
• not have to query the credit service since it has a view of the changes the credit service made to its model

Note that the latter option (not querying, just depending on the enriched view) decouples the two services.

The former option does allow the activity service to see state which was modified based on payment events after the enriched event's payment event: this may or may not be desirable.