Timestamps error while working with Prometheus Python client - "Error on ingesting samples that are too old or are too far into the future"

Question

I'm trying to import performance test results history to Prometheus and faced a strange issue with the official Python Prometheus client.

Such code works correctly:

dt_now = datetime.datetime.now(tz=pytz.timezone('UTC'))
gobj = GaugeMetricFamily('FooMetricGood', '')
gobj.add_metric([], 123, timestamp=dt_now.timestamp())
yield gobj

And such don't:

dt_format = '%Y-%m-%d_%H-%M-%S.%f %z'
dt_custom_str = '2021-11-11_18-12-59.000000 +0000'
dt_parsed_from_custom = datetime.datetime.strptime(dt_custom_str, dt_format)
gobj = GaugeMetricFamily('FooMetricNotWorking', '')
gobj.add_metric([], 789987, timestamp=dt_parsed_from_custom.timestamp())
yield gobj

I got warnings in Prometheus log like these:

prometheus-prometheus-1  | ts=2021-11-11T13:41:01.895Z caller=scrape.go:1563 level=warn component="scrape manager" scrape_pool=services target=http://192.168.64.1:8080/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

I tried to raise the issue in GitHub here with full working and not working code examples about 2 weeks ago but got absolutely no answers.

Any help will be very appreciated.

UPDATE : After answers and comments I've rechecked and here are some more details.

If I try two metrics from datetime now (see code example from GitHub link above) and datetime taken from string '2021-12-13_00-34-59.000000 +0000' all of them appear in Prometheus Python client web interface as:

# HELP FooMetricGood 
# TYPE FooMetricGood gauge
FooMetricGood 123.0 1639475119451
# HELP FooMetricGoodToo 
# TYPE FooMetricGoodToo gauge
FooMetricGoodToo 456.0 1639475119451
# HELP FooMetricNotWorkingNew 
# TYPE FooMetricNotWorkingNew gauge
FooMetricNotWorkingNew 789987.0 1639355699000

But in the log of the Prometheus server I see:

prometheus-prometheus-1  | ts=2021-12-14T09:51:35.524Z caller=scrape.go:1611 level=debug component="scrape manager" scrape_pool=services target=http://192.168.64.1:8080/metrics msg="Out of bounds metric" series=FooMetricNotWorkingNew
prometheus-prometheus-1  | ts=2021-12-14T09:51:35.524Z caller=scrape.go:1563 level=warn component="scrape manager" scrape_pool=services target=http://192.168.64.1:8080/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

As far as I understand in Prometheus Python timestamps are in milliseconds so I've compared them

FooMetricGood 1639475119451
FooMetricNotWorkingNew 1639355699000

and got:

1639475119451 - 1639355699000 = 119420451 milliseconds = (119420451 / 1000 / 60 / 60) hours = 33.1723475

So according to Prometheus Python current time is only 33 hours after the bad metric timestamp.

I tried to tweak the date and make it 2021-12-14_08-34-59.000000 +0000, now difference is only 1.2913288888888887 hours before the present time but still not working.

Answer 1

I was able to successfully execute your code and query the metric in prometheus: https://replit.com/@pygeek1/ComplicatedAcidicAxis#main.py

I suspect that there is an issue with the Prometheus server. Ensure server time is set correctly. A similar issue was reported over a 1 year ago, caused by an issue regarding server time constantly changing: https://github.com/prometheus/prometheus/issues/6554