Matplotlib/seaborn heatmap with a datetime x-axis shows 1970 instead of real datetime with AutoFormatter

Question

I have a heatmap produced with the following pivot table:

HCpredictedLabel         B     H      R
datetime                               
2021-07-20 09:00:00  115.0  80.0  119.0
2021-07-20 09:05:00   69.0  38.0  149.0
2021-07-20 09:10:00   58.0  50.0  131.0
2021-07-20 09:15:00   71.0  31.0  162.0
2021-07-20 09:20:00   78.0  38.0  164.0
...                    ...   ...    ...
2021-07-21 07:35:00    3.0  10.0   14.0
2021-07-21 07:40:00    9.0   1.0   30.0
2021-07-21 07:45:00    8.0   3.0   31.0
2021-07-21 07:50:00   12.0  13.0   26.0
2021-07-21 07:55:00   25.0  35.0   97.0

[114 rows x 3 columns]

The x-axis of the heatmap shows the datetime, which I would like to be automatically scaled, depending on the range in the dataset (can be days, weeks, or even months).

Here is my current script:

import seaborn as sns
import datetime as dt
import matplotlib.pyplot as plt
import matplotlib.dates as mdates    
from matplotlib.dates import AutoDateFormatter, AutoDateLocator

plot_df.fillna(0,inplace=True)
fig, axHM = plt.subplots(1,1,figsize=(30,5))

axHM = sns.heatmap(plot_df.T, cmap='rocket')
locator = mdates.AutoDateLocator()
formatter = mdates.AutoDateFormatter(locator)

axHM.xaxis.set_major_locator(locator)
axHM.xaxis.set_major_formatter(formatter)
fig.autofmt_xdate()

I get this plot, with the x-axis showing dates in a nice format but at a completely wrong scale and from 1970, rather that my datetime dates from the table:

Any help appreciated! Thank you