Batch operation over thousands of aggregates in CQRS system, do people do that?

Question

I'm working on an application for managing bank credit cards. CQRS and Event Sourcing architecture was chosen for the app. The most important aggregate in the app is CreditCard which controls the credit card lifecycle. It looks something like:

class CreditCard {
   private int status;
   public void activate() {...}
   public void deactivate () {...}
   ...
}

Its activate and deactivate methods protects credit card invariants and publish CardActivatedEvent and CardDeactivatedEvent, respectively, if the invocation of the method succeeds. We store these events in the event store for later aggregate reconstruction on the command side. We apply these events to various views. We use these events to notify other third party systems. All good for now.

Recently, we got a new requirement to charge all active credit cards on monthly basis.

My first instinct was, ok we can add charge method to the same CreditCard aggregate. This method can check some invariants relevant to charging. Like, is the card in correct status for charging, was it charged already, etc. On successful invocation, this method can publish CardChargedEvent. Then we can create some process manager which will once per month query view side for active credit cards to get their IDs. Having these IDs, the process manager can issue multiple charge commands (one per credit card aggregate) to the command side. For each charge command received, the command side will reconstruct CreditCard aggregate object and call it's charge method. The only problem is that this approach looks quite inefficient. Especially regarding database roundtrips on the command side (one read and one write per aggregate instance). If we take into the equation that we can easily have 100k plus credit cards in our app, this roundtrip overhead starts looking to me as a bit of a problem.

Does anyone have any experience with batch operations on CQRS/ES systems? Is my concern valid? What to do in such cases? How you implement batches in CQRS systems?

One alternative that pops to my mind is that for charging use case I ditch CQRS/ES/DDD principles, and implement the whole thing using stored procedures on one of our view databases. This procedure can search for suitable credit cards in the credit card view table and populate the "to be charged queue" table with records found. Then I can have some external process that reads this second table and do whatever it needs to do.

Answer 1

Recently, we got a new requirement to charge all active credit cards on monthly basis. My first instinct was, ok we can add charge method to the same CreditCard aggregate.

I think this is where the design flaw happens.

Your CreditCard aggregate was designed with a specific use case in mind:

The most important aggregate in the app is CreditCard which controls the credit card lifecycle.

Charging the credit card is not part of the credit card lifecycle. Whether it happens or not depends on the credit card state, but charging (successfully) the credit card will not change the state of your domain object. It should not interact with the CreditCard domain aggregate, as the purpose of your aggregate is to enforce business rules when changing your state. You should ask yourself: what aggregate is changed when charging a credit card ?

The answer to this question depends on the rest of your domain model and business cases, but it has more to do to with stuff like account balance or credit authorization than the card itself. You could implement like this:

• A batch process working monthly would query your CreditCard aggregate for active cards, then try to charge all customers for their monthly fees by sending a command to the AccountBalance aggregate ;

• The AccountBalance aggregate would raise a BalanceChangedEvent if the customer has enough money, or a CreditAuthorizationRequiredEvent if not, temporarily freezing the account until the credit was authorized or rejected ;

• A CreditAuthorization aggregate could either allow or deny the credit, based on credit allowance business rules, raising events accordingly ;

• The AccountBalance aggregate would unfreeze the account, changing the balance or not based on the outcome, eventually raising or not the BalanceChangedEvent ;

• The CreditCard aggregate would register to the CreditDeniedEvent to deactivate the credit card because the customer was not able to pay the fees ;

... and so on ...