StarBase Blog

 

New Noise Testing Feature of the New LoadRunner 12.02

In my last blog article, HP LoadRunner 12.02 – First Look at the New Features, I said that the new Noise Testing feature available in the new HP LoadRunner 12.02 release was somewhat disappointing. In fact I went on to say that I thought it was a largely pointless enhancement, and that HP should have used their development time on useful features.

Well, Shane Evans (@ShaneEvansLR) of HP has been in touch and it looks like I may have been a little harsh on this feature.

The point I was making was that as it is so easy to record a single URL in VuGen, why would HP spend development time adding this as a feature.

What I now know from Shane is HP are positioning Noise Testing to cater for the situation in many applications where 85% of users just hit the homepage and don’t do anything useful i.e. they are just background noise.

The good thing, and something that wasn’t clear from the release information, is that recreating this noise will effectively be ‘free’, i.e. no virtual user license will be consumed – thus allowing the full license to be used for the actual business processes you want to test.

Setting up Noise Testing couldn’t be simpler – add a new group in LoadRunner and select the ‘Use a Noise Generator…’ option, the specify the URL and the number of vusers.

HP LoadRunner 12.02 Moise Generator

HP LoadRunner 12-02 Noise Generator image

When you run the performance test scenario each noise generator vuser will hit the URL as frequently as it can. If you want to change the pace of how often the URL is hit you can do this via the pacing options in the runtime settings.

StarBase Logo

StarBase – Performance Testing Specialists

So, based on the fact you can generate this background noise simply, and without using up your virtual user license, I am revising my opinion of Noise Testing from ‘largely pointless’ to ‘a good addition that demonstrates HP’s determination to make its LoadRunner technology the performance testing tool of choice’.

Alan Moulsdale, Technical Director, StarBase

HP LoadRunner 12.02 has just been released and to ease my way into 2015 I have been taking a look at its new features, three of which I would like to highlight…

Firstly there is the new web controller which has a ‘tech preview’ in HP LoadRunner 12.02 – tech preview is the new way of saying ‘beta’, i.e. the functionality is there but it is not officially supported and problems or issues you find won’t necessarily be fixed straight away.

The Web Controller is going to be shared by LoadRunner and Performance Center and I must say HP have done a good job. It has a nice clean, intuitive interface, and its graph handling and manipulation is excellent. I particularly like the ability to maintain a list of favourite Load Generators.

LoadRunner Web Controller

LoadRunner Web Controller

Secondly are the enhancements to the TruClient protocol’s Object Identification. Anybody that has used TruClient knows that it works brilliantly for 80% of web applications, 80% of the time. However if you are in that unfortunate 20:20 situation the inbuilt automatic object identification sucks! To get TruClient to work here used to require a rather in-depth understanding of XPath, and more importantly how to construct a syntactically correct XPath query – not easy. But not anymore. The 12.02 enhancements introduce ‘Descriptors’ and the Descriptor Editor. In this you select the attributes you want to identify the object by and TruClient does the rest. No XPath required.

TruClient protocol’s Object Identification
TruClient protocol’s Object Identification

And thirdly is Noise Testing. Now this sounded interesting when I read about it “New noise testing capabilities, allowing you to run a simple noise test alongside standard Vuser scripts. The noise test performs basic load testing without an actual business process.”, however the reality is somewhat disappointing. You tell LoadRunner the static URL you want to hit to generate ‘noise’ and LoadRunner will create a Vuser that hits it throughout your scenario. I know HP are trying to appeal to the masses to get universal acceptance for LoadRunner and other tools, but really, this is largely pointless enhancement and they could have spent the development time on useful features.

Noise Testing
Noise Testing

12.02 also contains all the usual protocol and technology updates you’d expect, and it is worth taking the time to install.

StarBase Logo

StarBase – Performance Testing Specialists

Alan Moulsdale, Technical Director, StarBase

 

 

The testing is finished and the client is happy

HP-LoadRunner-just-got-even-better

SAP BPC Performance Testing with LoadRunner

The testing is finished and the client is happy. As well as a good Subject Matter Expert we also had the assistance of a very good SAP VTO (Volume Test Optimisation) Team. The team consisted of 2 people, one looking at BPC itself and one looking at the database server. They are able to log in as an SAP support team and hence have access to everything that SAP provides as well as to internal SAP resources.

Between our team, running the scripts and reporting on the response times, the Subject Matter Expert and the VTO team we were able to tune the application to cut down on the report response times and we got as far as ensuring that none of them timed out and we cut the response times down by about 50 percent over two weeks.

There were some SAP Notes that needed implementing. We also found that whilst they never actually run out the CPU and memory availability levels were getting very low, they’ll need to be increased before the second region goes live but that gives us a number of months to plan it properly. The VTO Database Guru also suggested some extra indexes that he felt would help.

We also found that the transparent “hot” backup that took place at 2400CET was anything but transparent. The service provider is now looking at this but for now the scheduled backup time has been adjusted to be outside the user working hours. There is also a daily consolidation job that needs to be run to ensure that the reports can be run as efficiently as possible. This is taking forty minutes instead of the expected five so that is currently being worked on.

All in all, a very successful test execution phase, we’ve since rerun our main test after the VTO recommendations had been implemented and gained more improvement to the response times, in the region of a 50% improvement in most cases.

Read Part 6 here


Lessons Learned

Lessons learned? A number really

HP-Gold-Partner-StarBase

StarBase – HP Gold Partner

  1. The scripts are large but need very little correlation. Also parameterising anything other than some basic values will be very difficult. The Subject Matter Expert had built exactly the same reports where the only changes were to some of the dimensions. Each set of dimensions returned the same amount of data, rows and columns in the spread sheets. Parameterising to the extent of changing the number of data values being returned (rather than just the data values being returned) would have been almost impossible as the data definitions are also part of the message. It did mean that the effects of report sizes on the response times could not be measured but it was considered that this trade off was acceptable.
  2. The iteration execution time means thinking outside the bounds of regular performance testing as an iteration can take hours to execute. It’s quite common for someone, in our environment, to spend all day on what is really one Business process. In future cases it may be worth investigating the implication of dealing with each different report as a separate script and designing the testing around individual report execution frequencies rather than end to end processes.
  3. The need for production data volumes was really reinforced, this testing would have been much less useful without the full production data volumes.
  4. The data values need to be parameterised otherwise the server will not perform the updates, it can recognise when the values are the same as they were previously even if a message is being sent . We initially thought that leaving the values the same would still trigger an update as the server would just see a number of values, this wasn’t true. We found we really needed to change most of the values, rather than just one or two, to get the correct update profiles.
  5. BPC runs well at low levels of free server resources so you need server monitoring to determine the impact on server resources.
  6. As you would perhaps expect in this type of process, the correct indexes are very important. Perhaps not so obvious is that not implementing all available SAP Notes can have a major impact on response times.
  7. Perhaps an expensive option ( I never saw the figures) but bringing in a SAP VTO team can give big benefits in respect of getting access to SAP internal resources and effective tuning recommendations.
    StarBase Logo

    StarBase – Performance Testing Specialists

 

It’s execution time!

HP-Gold-Partner-StarBase

StarBase – HP Gold Partner

We’d not really thought about it whilst we were creating the scripts and in Vugen we were running them with think time turned off. Put the scripts in the controller and one script took over 3 hours to run one iteration! Our first test is to run each script in turn, 3 iterations – this was going to take over 18 hours for 6 scripts!

We discussed this with the Subject Matter Expert and cut the think times down to 120 seconds to read a report and 180 seconds to do the input, it increases the load but in the “safe” direction, we just cut the number of users a bit. In fact we trimmed most of the scripts so that they each ran in less than 30 minutes per iteration.

This raised another issue, nothing to do with BPC or LoadRunner, our RDP based injectors close their sessions (close rather than disconnect) after an hour of inactivity – which seems to be interpreted as no keyboard input – so running un-monitored tests was not practical. We’re getting this changed for the next testing cycle but this cycle will be completed before the CR gets actioned (update: the change was made in 2 days rather than the 15 of the SLA so we don’t have to check the injectors manually any more).

The main aim of our testing is to measure the report refresh times. The Subject Matter Expert has a big fear that too many requests will cause time outs. By default a time out is after 5 minutes – 5 minutes from submission so multiple users requesting the same report may create queues so some requests may not even get started before they time out. We actually found one process that timed out during our recording process so that was the first issue identified before we even got started in our testing phase: always nice to prove your worth early. A configuration change fixed this but it turned out that all testing up until then had been on sub 100 row data requests, we were at full power with several hundred thousand rows (an interesting question to muse upon: is 400,000 a few hundred thousand or several hundred thousand?)!

More Disks Please!

The data loading also resulted in us running out of disk space so more disks were ordered. This would have manifested itself within the production go-live cycle so there’s another good reason to performance test with production volumes. The Subject Matter Expert was rather pleased to have found that one before go-live, the three week lead time on new disk requests would have been very embarrassing and would probably have led to a delay in go-live.

The infrastructure is hosted and we’re not allowed access to the server performance metrics other than by requesting it on a daily basis, and then it’s only reported in 15 minute intervals. This has limited our monitoring and reporting, we’re strictly limited to reporting the response times, and change in response times due to load. This, sort of, makes the analysis easy (there’s not much to analyse) but makes any diagnosis of issues difficult, we’ve got a good Subject Matter Expert though and he’s on top of all that stuff, BPC itself seems to give him everything that he needs. We’re also running a session with SAP’s VTO team and they should be able to give us a few pointers to performance tweaks.

Part 5 will discuss our testing, results, conclusions and what we’ve learnt about BPC testing. We need to finish the testing first though!

 

StarBase Logo

StarBase – Performance Testing Specialists

Read Part 5 here


Client Testimonials