In last 2.5yrs of performance testing, I have come across some reports that only report an average response time (also during discussion have noticed people only mentioning average response time). For example, consider following two response time data set(graph and dataset). Also assume that the SLA is 6 seconds.
The average response time for both the dataset is 5 and therefore satisfies the SLA. However, looking at the data points you will find that the variation from the mean in DataSet1 (SD=3.19) is more than that in DataSet2 (SD=0.71). You have a mininum value of 1 and maximum value of 9 for dataset1 and you have minimum of 4.5 and maximum of 6 for dataset2. Also 4 out of 10(40%) response time values for dataset1 are above the expected 6 second target. Therefore there is somekind of performance anomoly. This kind of information is missed out if you only report average and therefore may lead into making an incorrect decision by stakeholder(s).
If possible, avoid using only average. However, if you don’t have an option then along with average, I suggest you also report minimum, maximum, standard deviation and sample size. It provides bit more information about the response time.
Update: Only thing that is reported is average response time and there is no graph to show the average response time over time
3 comments:
What I would say on top of this Harinder is that when you see the eratic behaviour of a minimum of 1 and a maximum of 9 although the SLA is met you can be sure that as you cale up the load, this is the area of the application that is going to start to bottleneck. This type of behaviour can tell us a few things, promarily.
Either the application is not very stable in terms of performance yet
Or we are executing a SQL query each iteration that is returning vastly different datasets in terms of their size.
I do not disagree with you, however what I wanted to prove was that if you don't have a graph and other measurements(stdev,min,max,etc)then just reporting an average response time could be a dangerous thing as illustrated using two data sets.I should have been bit more clearer.
I have updated by Blog now.
Post a Comment