Monday, January 10, 2011

Avoid reporting just an average response time

In last 2.5yrs of performance testing, I have come across some reports that only report an average response time (also during discussion have noticed people only mentioning average response time). For example, consider following two response time data set(graph and dataset). Also assume that the SLA is 6 seconds.



The average response time for both the dataset is 5 and therefore satisfies the SLA. However, looking at the data points you will find that the variation from the mean in DataSet1 (SD=3.19) is more than that in DataSet2 (SD=0.71). You have a mininum value of 1 and maximum value of 9 for dataset1 and you have minimum of 4.5 and maximum of 6 for dataset2. Also 4 out of 10(40%) response time values for dataset1 are above the expected 6 second target. Therefore there is somekind of performance anomoly. This kind of information is missed out if you only report average and therefore may lead into making an incorrect decision by stakeholder(s).

If possible, avoid using only average. However, if you don’t have an option then along with average, I suggest you also report minimum, maximum, standard deviation and sample size. It provides bit more information about the response time.

Update: Only thing that is reported is average response time and there is no graph to show the average response time over time

Wednesday, January 5, 2011

Siebel Loadrunner scripting issues

If you haven't scripted a siebel application before for load testing using Load Runner then you are in for some good time. I am sure you will come across different issues which you might not have encountered before. I wanted to compile a list of issues that I/colleague have encountered and how they were solved. Therefore this blog contains some of those issues and possible solution.

Following issues were encountered by my colleague:

Issue:
Sometimes the application does not function correctly when recording with Loadrunner, but works fine without recording.
Solution:
Close all browser instances, search and delete a file called Siebel High Interactivity Framework, Log into the Siebel application and agree to 'repair' the browser configuration.


Issue:
Some "web_url" requests fail during replay and the application responds with "We detected an Error which may have occurred for one or more of the following reasons: We are unable to process your request. This is most likely because you used the browser BACK or REFRESH button to get to this point.(SBL-UIF-00335)".

The following request is a sample which generated this error:

web_url("start.swe_54",
"URL=https://{Host}/{Env}/start.swe?SWECmd=GetCachedFrame&SWEC={Siebel_SWECount}&SWEFrame=top._sweclient._swecontent._sweview._swetmp&SWESP=1",
"TargetFrame=",
"Resource=0",
"RecContentType=text/html",
"Referer=https://{Host}/{Env}/start.swe?SWECmd=ShowPopupFrames&SWEDIC=1&SWEU=/{Env}/start.swe%3fSWECmd%3dGetCachedFrame%26SWEC%3d9%26SWEFrame 3dtop._sweclient._swecontent._sweview._swetmp%26SWESP%3d1",
"Snapshot=t62.inf",
"Mode=HTML",
LAST);

Solution:
At first glance the requested URL seems to be parametrized correctly. However, the automatically correlated SWEC={Siebel_SWECount} is actually incorrect. In most cases the Siebel_SWECount parameter is incremented correctly by Loadrunner. However, when using SWECmd=GetCachedFrame, that value for the SWEC parameter must be correlated manually from the previous request.

The response from the previous request will contain something like:

URL .../xxx/start.swe?SWECmd=GetCachedFrame&SWEC=32&.......

Normally this kind of correlation is natural and intuitive, but with the Siebel protocol SWEC is a counter which is not usually correlated and therefore easily overlooked.


If you come across different issue(s) related to Siebel application scripting, I would like to hear about them as well as how you were able to solve it. If I come across more issues, I will update this blog.

Sunday, December 12, 2010

LoadRunner script for SOAP over JMS to test TIBCO Middleware

Following is a script that I created to test SOAP message over JMS using SOA protocol. I would suggest using JAVA protocol to script SOAP over JMS rather than SOA protocol. It provides greater flexibility than SOA protocol. There is a good blog written by Stuart Moncrieff and I suggest reading it if you are planning on using JAVA protocol to test SOAP message over JMS. I don't want to reinvent the wheel by writing a new code. You will require some modification to the Java script. For example, if you are using username and password, you will need to pass these values into the createConnection function.

It will look like this(It is modification of Stuart's code):
queueConnection = queueConnectionFactory.createConnection(username,password);

Before you starting writing your script using either SOA or Java protocol, you will need few files and information and they are:

1: Tibjms.jar -compulsory
2: Java 1.6 -compulsory (older version may work, but I tested against 1.6)
3: JNDI initial context factory -compulsory
4: JNDI provider URL - compulsory
5: JMS connection factory - compulsory
6: JMS security credentials - compulsory, if implemented
7: JMS security principal - compulsory, if implemented
8: Request queue name - required, if you are sending message to the request queue
9: Receive queue name - required, if you are receiving message from the receive queue

You will need to get information for points 3-9 from your JMS administrator or whoever may have implemented JMS.

SOA Protocol script
The script was meant to test TIBCO with JMS Fire and forget pattern. No response was expected from TIBCO because the test was to find out whether the request queue is able to handle the expected requests per hour. Therefore, I did not require Receive queue name details.

Following details were set in the Run-time setting-> JMS advance option. For points 2-5, I have used dummy names.

1:JNDI initial context factory :com.tibco.tibjms.naming.TibjmsInitalContextFactory //testing TIBCO JMS. This is a dropdown option and might change depending on what you are testing.
2:JNDI provider URL :tibjmsnaming://tim.tom.au:12345
3:JMS connection factory :QueueConnectionFactory
4:JMS security credentials :jmsPerTester
5:JMS security principal :jmsPerTester

SOAP request
char *EmployeeDetails=
""
""
"Tom"
""
"110 Latrobe st"
"Melbourne"
"3000"
"Vic"
"1234567890";

Action()
{
long currentTime=1; 
char *ExpiryTime; 

//get current time in seconds
currentTime=time(¤tTime); 

currentTime = currentTime + 500000; 

ExpiryTime = (char*)malloc(20, sizeof(char)); 

//save current time value into ExpiryTime variable. This is how long the message should be valid for
sprintf(ExpiryTime,"%d",currentTime); 

//Save EmployeeDetails SOAP request into LR parameter
lr_save_string(EmployeeDetails,"EmployeeDetailSOAP_param");

//lr_output_message ("%s",lr_eval_string ("{EmployeeDetailSOAP_param}"));


//setup JMS properties
jms_set_general_property("msgJMSMessageType","JMS_MESSAGE_TYPE","BytesMessage"); //Change 'ByteMessage' to 'TextMessage', if you are sending as a text
jms_set_message_property("msgJMSExpiration","JMSExpiration", ExpiryTime); 
jms_set_message_property("msgJMSMessageID","JMSMessageID", "-JMSMessageID-");
jms_set_message_property("msgJMSPriority","JMSPriority", "4"); 
jms_set_message_property("msgJMSRedelivered","JMSRedelivered", "false");
jms_set_message_property ("msgJMSDeliveryMode","JMSDeliveryMode","2");
jms_set_message_property("msgJMSCorrelationID", "JMSCorrelationID", "VuserID-{VUserID}_{TimeDate}"); //VUserID & TimeDate are loadrunner parameters


//sending message
lr_start_transaction ("SoapRequest"); 
jms_send_message_queue("step 1: Sending SOAP message","{EmployeeDetailSOAP_param}", "EmployeeDetail.request.queue.com.au");  


//Receive message
/*jms_receive_message_queue("step 2: Received SOAP message", "EmployeeDetail.response.queue.com.au"); 
lr_message(lr_eval_string("{JMS_message}")); */

lr_end_transaction ("SoapRequest",LR_AUTO);

free(ExpiryTime);
return  0;
}

Note:In the above code I am using lr_start_transaction function to capture total number of requests posted to the queue. It does not measure the response time. This is because we are not expecting any response due to Fire and Forget pattern.

You could use file(/char array) for the SOAP request as well as parametrize the SOAP content. I will leave this as an exercise for anyone who wants to try it out.

Tuesday, December 7, 2010

Setting JMS DeliveryMode in LoadRunner

I have been scripting SOAP over JMS using LoadRunner SOA protocol and had forgotten that LoadRunner uses value associated with Delivery mode "PERSISTENT" or "NONPERSISTENT"/"NON_PERSISTENT".

Below is the screenshot of what loadRunner Script code & log looked when I used "NONPERSISTENT" text as my parameter in jms_set_message_property function. If you don't know the Deliverymode value, you may think value "2" is for "NONPERSISTENT" mode but that is incorrect as seen in HermesJMS(see following screenshot).






After I updated the function by passing the correct DeliveryMode value, everything looked great as seen from LoadRunner log and HermesJMS.





If you want JMS delivery message to be "PERSISTENT" then you have to use value "2" and for "NONPERSISTENT" message you need to use value "1" in your jms_set_message_property function.

Therefore your jms_set_message_property function will look like this:

jms_set_message_property("MsgDeliveryMode","JMSDeliveryMode","1"); //NONPERSISTENT msg

jms_set_message_property("MsgDeliveryMode","JMSDeliveryMode","2"); //PERSISTENT msg

Thursday, December 2, 2010

SAXParseException JMSQueue

If your queue is set up to receive JMSMessage type as ByteMessage and you are sending TextMessage, you may receive SAXParseException message. See the screenshot below.



I have been using HermesJMS application to view Tibco JMS queues during SOAP over JMS scripting. It allowed me to fix issues that I encountered during scripting.

You can download HermesJMS application and follow TibcoJMS blog by Serge on how to connect HermesJMS to TibcoJMS queue.