Percentile values in dashboard are varying with the order of data in the input CSV

Question

We have a use case where we are merging CSV files from two different, independent JMeter instances and using the combined CSV file to generate the dashboard. While merging two files, there is a question about the order of the data points in the CSV and how it impacts the final dashboard. To test it out, I created two CSV files with the same data but one where the data is in increasing order of timestamps and the other is in decreasing order and compared the Response Time Percentiles Over Time (successful responses) graphs.

From what I understand from Kth select algorithm used for percentile calculation, if the input data set contains of the same values, irrespective of the order, the output should be the same every time. In JMeter, the input data set is controlled by the jmeter.reportgenerator.statistic_window property.

So, my hypothesis is, if the statistic_window is large enough to hold all the samples for a given time-granularity window, the output graphs should be 100% identical but that doesn't seem to be the case. I generated graphs with a statistic_window of 100 million which is far more than the number of samples in the entire CSV file and still, the graphs are different from one another, even if it is by a small percentage.

I like to understand if my assumptions are wrong or if there is a bug in JMeter that I need to search for.

Answer 1

Turns out there was a bug in JMeter which I was able to submit a patch for. For more details: https://bz.apache.org/bugzilla/show_bug.cgi?id=65446