Data fails to update in Redis on Azure

Question

In order to be able to load the order list pages of our application quickly, we are trying to push orders as soon as they come in to our application to Redis. This means multiple users creating multiple orders consecutively. On average 10 to 15 orders are created consecutively, while at the same time existing order are updated with new info or statuses.

We are trying to leverage Azure Redis caching where concurrent updates are expected at a high rate. We have a list object kept in a single redis key and whenever a user requests for the data to be shown in the list page, we will fetch the data from redis cache, apply necessary filters on the list data and provide the paginated output to the user. Code is in Dot Net Framework.

As mentioned above, data in redis is expected to be updated very frequently. So to make the update process sequential and to avoid any overwriting, we are using Azure Service Bus as the queueing mechanism. However, almost every time we update data in bulk, many of the records updated go missing and do not make it to Redis.

Following is the code of the queue consumer.

private async Task FetchFromQueue()
        {
            string controlNo = "";
            if (receiver == null || (receiver != null && receiver.IsClosedOrClosing == true))
            {
                receiver = new MessageReceiver(ServiceBusConnectionString, ServiceBusQueueName, ReceiveMode.PeekLock);
                receiver.PrefetchCount = 10;
                // Send and Receive messages with prefetch ON
                var receivedMessage = await receiver.ReceiveAsync(TimeSpan.FromMinutes(5));
                if (receivedMessage != null)
                {
                    controlNo = receivedMessage.Label.ToString();
                    await Task.Run(() => SaveToRedisFromQueue(controlNo, isInsert));
                }
                while (receivedMessage != null)
                {

                    await receiver.CompleteAsync(receivedMessage.SystemProperties.LockToken);
                    receivedMessage = await receiver.ReceiveAsync(TimeSpan.FromMinutes(5));
                    if (receivedMessage != null)
                    {
                        controlNo = receivedMessage.Label.ToString();
                        await Task.Run(() => SaveToRedisFromQueue(controlNo, isInsert));
                    }

                }
                if (receivedMessage != null)
                    await receiver.CompleteAsync(receivedMessage.SystemProperties.LockToken);

                await receiver.CloseAsync();
            }
        }
Below mentioned is the code for updating data to redis.

private async Task SaveToRedisFromQueue(string controlNo, bool isInsert = false)
        {
                List<OrderListOutputModel> lstCacheOrder = await _azureRedisCache.GetDataFromRedisAsync<List<OrderListOutputModel>>(cacheName);
                if (lstCacheOrder == null)
                {
                    await this.InsertOrderInfoToRedisCache();
                    lstCacheOrder = await _azureRedisCache.GetDataFromRedisAsync<List<OrderListOutputModel>>(cacheName);
                }

                var indexOf = lstCacheOrder.IndexOf(lstCacheOrder.Find(x => x.ControlNo == controlNo));
                        SqlParameter[] spParams = new SqlParameter[] {
                             new SqlParameter { ParameterName = "ControlNumber", Value = !string.IsNullOrEmpty(controlNo) ? Convert.ToInt64(controlNo) :0 }
                             };

                        OrderListOutputModel order = _iUnitOfWork.OrderRepository.ExecWithStoreProcedureGetFirstOrDefault<OrderListOutputModel>("Cache_GetOrderInfoByControlNumber", spParams);

                        if (order != null)
                        {
                            if (indexOf == -1)
                            {
                                //If data not present in cache add it
                                lstCacheOrder.Add(order);
                                await _azureRedisCache.InsertDataToRedis(cacheName, lstCacheOrder);
                            }
                            else
                            {
                                //If data not present in cache update it
                                lstCacheOrder[indexOf] = order;
                                await _azureRedisCache.InsertDataToRedis(cacheName, lstCacheOrder);
                            }
                        }
}

After placing some logs, came to understanding that some messages are not being consumed from the queue. Also some messages which were consumed were still missing from cache.

Could anyone please let me know what are we doing wrong here?

Answer 1

However, almost every time we update data in bulk, many of the records updated go missing and do not make it to Redis.

This issue could be due to Redis cache is currently unable to accept the update request, busy processing a previous update request or is a system maintenance which means cache has failed and further cache management operations are blocked until the cache status is restored.

You can refer to Azure Cache for Redis monitoring and troubleshooting FAQs, Troubleshoot Azure Cache for Redis server issues and https://docs.microsoft.com/en-us/answers/questions/597862/error-trying-to-scale-azure-cache-for-redis.html