AWS Kinesis "DynamicPartitioning.ActivePartitionsLimitExceeded" Error

Question

Solution Overview

An application publish events to a SNS topic, and a Kinesis delivery stream (Firehose) subscribes to the topic, and delivers events directly (no lambda) to a S3 bucket. A glue table is defined on the bucket so that we can query the table using Athena.

Problem

In the Firehose delivery stream we configured Dynamic Partitioning (DP) and the key is the primary key of the event record. It was working fine. However when we publish 10K events within less than 10 seconds, we got this DynamicPartitioning.ActivePartitionsLimitExceeded error and lots of events were not saved into the bucket with proper prefix.

I tried turning off DP but got "AppendDelimiterToRecordProcessor can only be present when Dynamic Partitioning is enabled" error. After I removed this processor, all the events ended up stored in one file without proper delimieter and Athena could only recognise the first one.

Question

I am new to Kinesis, but I thought Kinesis delivery stream + Athena should work together very well. It seems to me that without DP it would not work? I am wondering how people used them before DP was introduced late last year?

AWS doc does explain this, but I am just wondering if Kinesis Firehose + Athena will work without DP? We don't really need DP ourselves.

Update

My problem is similiar to the following, but I don't get a chance to transform the event when the source of the Firehose is a SNS topic, also I don't want to write a lambda to transform the data yet.

Kinesis/Firehose/Athena - Creating queryable data from logs

Firehose to S3 with One Record Per Line