TeamCity seems to believe the mysql driver isn't there during an update [ No suitable JDBC driver found]

Question

My TeamCity is down because of an update downloaded and initiated From TeamCity UI because it cannot find a suitable jdbc driver for MySQL.

we are using:

• TeamCity v2022.04 (build 108502)
• Mysql mysql Ver 14.14 Distrib 5.7.35-38, for debian-linux-gnu (x86_64) using 6.3
• Ubuntu 16.04.

The mysql connector used for far throughout 2021 is : mysql-connector-java-8.0.16.jar located the at /opt/teamcity-datadir-2021-06-14/lib/jdbc/.

I have updated it to 8.0.29 hoping that it will be suitable enough for TeamCity but no luck.

cicd:~# ls -lha /opt/teamcity-datadir-2021-06-14/lib/jdbc/
total 2.5M
drwxr-x--- 2 teamcity teamcity 4.0K May  5 14:01 .
drwxr-x--- 3 teamcity teamcity 4.0K May 14  2021 ..
-rw-r--r-- 1 teamcity teamcity 2.5M May  5 12:59 mysql-connector-java-8.0.29.jar 

Both INFO and DEBUG logs are pointing to the fact that the driver cannot be found but it's there. I have checked the permission, the owners it doesn't seem to want to pickup the driver at the location. We have been doing this level of backup for a long time and this is the first time this pops up.

Below is the server log

[2022-05-05 13:30:31,539]   INFO -  jetbrains.buildServer.STARTUP -    ===========================================================
[2022-05-05 13:30:31,540]   INFO -  jetbrains.buildServer.STARTUP - Starting TeamCity server
[2022-05-05 13:30:31,697]   INFO -  jetbrains.buildServer.STARTUP - TeamCity version: 2022.04 (build 108502), data format version 980
[2022-05-05 13:30:31,701]   INFO -  jetbrains.buildServer.STARTUP - OS: Linux, version 4.4.0-210-generic, amd64, Current user: teamcity, Time zone: UTC
[2022-05-05 13:30:31,702]   INFO -  jetbrains.buildServer.STARTUP - Java: 10.0.2, OpenJDK 64-Bit Server VM (10.0.2+13-Ubuntu-116.04.6, mixed mode), OpenJDK Runtime  Environment (10.0.2+13-
Ubuntu-116.04.6), Oracle Corporation; JVM parameters: --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=A
LL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL- UNNAMED --add-opens=java.rmi/sun.rmi.transp
ort=ALL-UNNAMED - Djava.util.logging.config.file=/opt/teamcity/2020.2.4/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xmx1024m -Dteamcity.configuration.path=../conf/teamcity-s
[2022-05-05 13:30:31,751]   INFO -  jetbrains.buildServer.STARTUP - The TeamCity Data Directory path is determined using the environment variable TEAMCITY_DATA_PATH
[2022-05-05 13:30:31,758]   INFO -  jetbrains.buildServer.STARTUP - The TeamCity Data Directory "/opt/teamcity-datadir-2021-06-14" exists
[2022-05-05 13:30:31,766]   INFO -  jetbrains.buildServer.STARTUP - Node specific data directory location: /opt/teamcity-datadir-2021-06-14
[2022-05-05 13:30:31,766]   INFO -  jetbrains.buildServer.STARTUP - Will use custom path for the database properties file: /opt/teamcity-datadir-2021-06-14/config/database.properties
[2022-05-05 13:30:31,767]   INFO -  jetbrains.buildServer.STARTUP - The TeamCity server caches directory is set to the path: /opt/teamcity-datadir-2021-06-14/system/caches
[2022-05-05 13:30:31,769]   INFO -  jetbrains.buildServer.STARTUP - The main configuration file "/opt/teamcity-datadir-2021-06-14/config/main-config.xml" exists
[2022-05-05 13:30:31,769]   INFO -  jetbrains.buildServer.STARTUP - The database properties file "/opt/teamcity-datadir-2021-06-14/config/database.properties" exists
[2022-05-05 13:30:31,799]   INFO -  jetbrains.buildServer.STARTUP - The internal database data file "/opt/teamcity-datadir-2021-06-14/system/buildserver.data" doesn't exist
[2022-05-05 13:30:31,805]   INFO -  jetbrains.buildServer.STARTUP - Data Directory version: 976
[2022-05-05 13:30:31,807]   INFO -  jetbrains.buildServer.STARTUP - Server internal properties (1): rest.cors.origins=https://login.microsoftonline.com
[2022-05-05 13:30:31,808]   INFO -  jetbrains.buildServer.STARTUP - TeamCity main node (id: MAIN_SERVER) is starting...
[2022-05-05 13:30:31,808]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Looking for the database configuration
[2022-05-05 13:30:31,809]   INFO -  jetbrains.buildServer.STARTUP - Database properties file "/opt/teamcity-datadir-2021-06-14/config/database.properties" exists
[2022-05-05 13:30:31,809]   INFO -  jetbrains.buildServer.STARTUP - Internal HSQL database file (/opt/teamcity-datadir-2021-06-14/system/buildserver.data) doesn't exist
[2022-05-05 13:30:31,809]   INFO -  jetbrains.buildServer.STARTUP - Database connection URL: "jdbc:mysql://localhost/teamcity_2021_06_14"
[2022-05-05 13:30:31,810]   INFO -  jetbrains.buildServer.STARTUP - Using database connection URL from the database properties file. The URL is:   "jdbc:mysql://localhost/teamcity_2021_06_14"
[2022-05-05 13:30:31,812]   INFO -  jetbrains.buildServer.STARTUP - Using external (MYSQL) database
[2022-05-05 13:30:31,812]   INFO -  jetbrains.buildServer.STARTUP - Current stage: Connecting to the database
[2022-05-05 13:30:31,817]   INFO -  jetbrains.buildServer.STARTUP - JDBC loader: loading JDBC drivers (for the first time)
[2022-05-05 13:30:31,907]  ERROR -  jetbrains.buildServer.STARTUP - No suitable JDBC driver found for database type: MySQL
[2022-05-05 13:30:31,907]   INFO -  jetbrains.buildServer.STARTUP - Current stage: TeamCity server startup error
[2022-05-05 13:30:31,907]   INFO -  jetbrains.buildServer.STARTUP - Administrator can login from web UI using super user authentication token
[2022-05-05 13:30:31,908]   INFO -   jetbrains.buildServer.SERVER - Super user authentication token: xxxxxxxxxx (use empty username with the token as the password to access the server)

The debug log

[2022-05-05 14:12:21,029]  DEBUG -  jetbrains.buildServer.STARTUP - Setting up database manager using the existent 'database.properties' file
[2022-05-05 14:12:21,066]  DEBUG -  jetbrains.buildServer.STARTUP - Attempt to connect to the database
[2022-05-05 14:12:21,069]  ERROR -  jetbrains.buildServer.STARTUP - No suitable JDBC driver found for database type: MySQL
[2022-05-05 14:12:21,070]  DEBUG -  jetbrains.buildServer.STARTUP - No suitable  JDBC driver found for database type: MySQL
jetbrains.buildServer.maintenance.exceptions.StartupDatabaseAccessingError: No suitable JDBC driver found for database type: MySQL
    at  jetbrains.buildServer.maintenance.StartupProcessor.performConnectToDatabaseInternal(StartupProcessor.java:712) ~[web.jar:?]
    at  jetbrains.buildServer.maintenance.StartupProcessor.performConnectToDatabase(StartupProc essor.java:23) ~[web.jar:?]
    at jetbrains.buildServer.maintenance.StartupProcessor.doDatabaseConnect(StartupProcessor.java:22) ~[web.jar:?]
    at  jetbrains.buildServer.maintenance.StartupProcessor.access$800(StartupProcessor.java:734) ~[web.jar:?]
    at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:5) ~[web.jar:?]
    at  jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:91) ~[common-api.jar:108502]
    at  jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:441) ~[web.jar:?]
    at  jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProc essor.java:87) ~[web.jar:?]
    at  jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProc essor.java:65) ~[web.jar:?]
    at  jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:1475) ~[web.jar:?]
    at  jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:1) ~[web.jar:?]
    at java.lang.Thread.run(Thread.java:844) ~[?:?]
Caused by: jetbrains.buildServer.serverSide.db.DatabaseDriverMissingException: No suitable JDBC driver found for database type: MySQL
    at  jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.connect(TeamCityDatabaseManager.java:610) ~[db.jar:?]
    at jetbrains.buildServer.maintenance.StartupProcessor.performConnectToDatabaseInternal(StartupProcessor.java:1116) ~[web.jar:?]

I am really not sure what's causing this confusion or how or where to set the path to the mysql connector for TeamCity to recognise. Anyone run into this before?

thanks in advance