'Debezium Oracle Connectors CDC usage on top of Kafka and Backup / DataGuard
We are trying to use the Oracle Debezium connector (1.9b) on top of Kafka. We tried to use 2 things regarding snapsot_mode: schema_only and initial. We use "log.mining.strategy":"online_catalog" (should be the default)
We are using a PDB/CDB Oracle instance on Oracle 19c. My understanding is that;
- The connector create a session to the PDB
- It add a shared lock to ensure the structure will not change (shared) for a short duration
- the DDL structure is retrieved from the PDB
- It create a session to the CDB
- It retrieve the last LSN event from CDB
- if snapshot == initial, it will use a "JDBC query to retrieve the whole data" from PDB
- it does NOT seems to release the initiated session (or rather process) to the PDB
- it continues to mines new events from CDB x. ... it seems to work for a couple of minutes
- After a couple of minutes, the number of process increase drastically
- The Oracle database freeze, due to an excess of process (that you can follow using v$process)
We had a lot of errors message; like:
A. Failed to resolve Oracle database
B. IO Error: Got minus one from a read call
C. ORA-04025: maximum allowed library object lock allocated
D. None of log files contains offset SCN: xxxxxxx
The message in point D. said it tries to use a offset which was part of "an old" archived log. Every 30min (or before, it we have more activity), the log is switched from a file to another. And a backup is occuring every 30minutes which will read the logs, backup it and then: delete it .
It seems to me that Debezium tried to reach past archived log whose was deleted by the backup process.
The process of "deleting previous archived logs" seems "correct" to me, isn't it ?
Why Debezium tries to pass through the archived logs ? because when snapshot==schema_only it should only catch the news events, therefore why using the archived one ?
How can we manage it ? I hope that if this point is resolved in my use-case, the debezium will stop to "loop" creating new process and ultimately will stop blocking the Oracle DB.
If you have any clues or opinions, don't hesitate to share it. Many thanks
We try to use shared lock and none We try to limite the number of tables in the scope
I cannot ask to stop the backup, in production it's not a good idea and in test, it seems that the backup is only there to clean the archived logs and avoid ending with completely used storage.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
