Web Application Load, Stress and Performance Testing Using WAPT

Why most of the manual testers fail when testing websites for performance? There are couple of reasons.
- They don’t have proper tools to test website for performance and
- They don’t have required skills for performance testing.

Does that mean you should wait till your stakeholder report the performance glitches in web application you developed? Definitely not. Many testers are good at testing websites manually and they report almost every defect while testing under standard tests. BUT, when same tester performs load or stress tests they stuck either at resource (required tools) or skill level.

I suggest not to take any risk if you are committed to defect free service. Ask for required tools and train your staff for necessary skills. Today, I’m going to review load, stress and performance testing tool for websites. The tool is called WAPT – Web Application Load, Stress and Performance Testing

WAPT allows you to perform website load and performance testing by creating heavy load from a single or multiple workstations. When you set and run your tests with this tool within a matter of minutes you can get performance report of your website or web application. WAPT uses powerful virtual users same as the real world users with full control over how to customize these virtual users.

Measuring website performance:

Did you ever wonder?
- How many users can work simultaneously on your website with acceptable quality of service?
- How many visitors your website can handle by day or hour?
- What is your website response time under load?

These all questions are nothing but the measure of website “performance characteristic”.

WAPT – website performance tool performs test by emulating activity of many virtual users. Each virtual user can have its own profile settings. You can have thousands of virtual users acting simultaneously on your website performing any activity like reading or writing with your web server. Once you set number of virtual users to act on your website you have option to run your tests for specified time or specified user sessions.

Analyzing the test report:
Test result consists of charts updated in real time which you can monitor when your tests are running. The final comprehensive report is provided at the end of the tests.

Here are the important parameters to be monitored on the test report:
Error Rate: Failure rate against total number of tests run. The error may be due to the high load on server or due to the network problems and timeouts.

Response Time: Obviously a great parameter to check when you run tests for website performance. This response time indicates time required by server to provide correct reply to the request.

Number of pages per second: Number of page requests successfully completed by server per second.
How to conclude performance tests?
These performance criteria change during each test-pass with different load conditions. You need to conclude what is your acceptable load limit and whether your server is able to serve this load.

E.g.: If you expect your server to handle 100 requests successfully per second then anything below this will be failure of your server which needs to be tackled.

How to Record tests:

WAPT works like any other record and playback tool but the real strength is behind it’s parametrization where you can configure any parameter from website url or user session to act as a real user.

Testing with WAPT in simple 5 steps:
Record->Configure->Verify->Execute->Analyze

WAPT uses inline Microsoft internet explorer which is used to record your interaction with website. When you record your test all dynamic parameters are recorded as static values which can be configured later while test execution. You then need to configure each user with different settings like unique sessions, number of virtual users, values for dynamic parameters etc. Once you done with recording and configuration just verify your test if it’s ready to run and then execute performance tests if everything looks ok. Finally analyze reports to decide website performance test as accepted or failed against your set of defined standards. That’s it.

WAPT is available in two versions
- Standard version (Latest WAPT 7.1)
- Professional version of this stress and performance testing tool (Latest WAPT Pro 2.1)

What WAPT Pro can do for you?
- Use several computers to generate load on website
- Measure web server performance in terms of CPU, RAM or network usage
- You can include the execution of a JavaScript code into virtual user profiles.

If you don’t want to specify every parameter manually you can use some technology specific modules to significantly improve your test experience.

Following additional modules can be downloaded and installed along with standard or professional version of WAPT:
- Module for ASP.NET testing
- Module for Adobe Flash testing
- Module for JSON format

Finally, any review can’t be complete without the list of Pros and cons.

WAPT Pros:

- Easy to install – Takes only 5 minutes to install
- Easy to use with very short learning curve
- You get run-time reports so that you can decide whether to continue the test or not, saving your big time.
- Detailed test report with graphical representation.
- Supports secure HTTPS protocol.
- 30 days free trial available!

WAPT Cons:

- Only windows platform supported to install this tool. (But you can test your website running under any OS and technology)
- No scripting ability
- It’s not free

How to try this tool?
You can download 30 day trial version of WAPT from here.

What is difference between Performance Testing, Load Testing and Stress Testing?

1) Performance Testing:

Performance testing is the testing, which is performed, to ascertain how the components of a system are performing, given a particular situation. Resource usage, scalability and reliability of the product are also validated under this testing. This testing is the subset of performance engineering, which is focused on addressing performance issues in the design and architecture of software product.

Performance Testing Goal:

The primary goal of performance testing includes establishing the benchmark behaviour of the system. There are a number of industry-defined benchmarks, which should be met during performance testing.

Performance testing does not aim to find defects in the application, it address a little more critical task of testing the benchmark and standard set for the application. Accuracy and close monitoring of the performance and results of the test is the primary characteristic of performance testing.

Example:

For instance, you can test the application network performance on Connection Speed vs. Latency chart. Latency is the time difference between the data to reach from source to destination. Thus, a 70kb page would take not more than 15 seconds to load for a worst connection of 28.8kbps modem (latency=1000 milliseconds), while the page of same size would appear within 5 seconds, for the average connection of 256kbps DSL (latency=100 milliseconds). 1.5mbps T1 connection (latency=50 milliseconds) would have the performance benchmark set within 1 second to achieve this target.

For example, the time difference between the generation of request and acknowledgement of response should be in the range of x ms (milliseconds) and y ms, where x and y are standard digits. A successful performance testing should project most of the performance issues, which could be related to database, network, software, hardware etc…

2) Load Testing:

Load testing is meant to test the system by constantly and steadily increasing the load on the system till the time it reaches the threshold limit. It is the simplest form of testing which employs the use of automation tools such as LoadRunner or any other good tools, which are available. Load testing is also famous by the names like volume testing and endurance testing.

The sole purpose of load testing is to assign the system the largest job it could possible handle to test the endurance and monitoring the results. An interesting fact is that sometimes the system is fed with empty task to determine the behaviour of system in zero-load situation.

Load Testing Goal:

The goals of load testing are to expose the defects in application related to buffer overflow, memory leaks and mismanagement of memory. Another target of load testing is to determine the upper limit of all the components of application like database, hardware and network etc… so that it could manage the anticipated load in future. The issues that would eventually come out as the result of load testing may include load balancing problems, bandwidth issues, capacity of the existing system etc…

Example:

For example, to check the email functionality of an application, it could be flooded with 1000 users at a time. Now, 1000 users can fire the email transactions (read, send, delete, forward, reply) in many different ways. If we take one transaction per user per hour, then it would be 1000 transactions per hour. By simulating 10 transactions/user, we could load test the email server by occupying it with 10000 transactions/hour.

3) Stress testing

Under stress testing, various activities to overload the existing resources with excess jobs are carried out in an attempt to break the system down. Negative testing, which includes removal of the components from the system is also done as a part of stress testing. Also known as fatigue testing, this testing should capture the stability of the application by testing it beyond its bandwidth capacity.

The purpose behind stress testing is to ascertain the failure of system and to monitor how the system recovers back gracefully. The challenge here is to set up a controlled environment before launching the test so that you could precisely capture the behaviour of system repeatedly, under the most unpredictable scenarios.

Stress Testing Goal:

The goal of the stress testing is to analyse post-crash reports to define the behaviour of application after failure. The biggest issue is to ensure that the system does not compromise with the security of sensitive data after the failure. In a successful stress testing, the system will come back to normality along with all its components, after even the most terrible break down.

Example:

As an example, a word processor like Writer1.1.0 by OpenOffice.org is utilized in development of letters, presentations, spread sheets etc… Purpose of our stress testing is to load it with the excess of characters.

To do this, we will repeatedly paste a line of data, till it reaches its threshold limit of handling large volume of text. As soon as the character size reaches 65,535 characters, it would simply refuse to accept more data. The result of stress testing on Writer 1.1.0 produces the result that, it does not crash under the stress and that it handle the situation gracefully, which make sure that application is working correctly even under rigorous stress conditions.

Database Testing – Practical Tips and Insight on How to Test Database

Database is one of the inevitable parts of a software application these days. It does not matter at all whether it is web or desktop, client server or peer to peer, enterprise or individual business, database is working at backend. Similarly, whether it is healthcare of finance, leasing or retail, mailing application or controlling spaceship, behind the scene a database is always in action.

Moreover, as the complexity of application increases the need of stronger and secure database emerges. In the same way, for the applications with high frequency of transactions (e.g. banking or finance application), necessity of fully featured DB Tool is coupled.

Currently, several database tools are available in the market e.g. MS-Access2010, MS SQL Server 2008 r2, Oracle 10g, Oracle Financial, MySQL, PostgreSQL, DB2 etc. All of these vary in cost, robustness, features and security. Each of these DBs possesses its own benefits and drawbacks. One thing is certain; a business application must be built using one of these or other DB Tools.

Before I start digging into the topic, let me comprehend the foreword. When the application is under execution, the end user mainly utilizes the ‘CRUD’ operations facilitated by the DB Tool.

C: Create – When user ‘Save’ any new transaction, ‘Create’ operation is performed.
R: Retrieve – When user ‘Search’ or ‘View’ any saved transaction, ‘Retrieve’ operation is performed.
U: Update – when user ‘Edit’ or ‘Modify’ an existing record, the ‘Update’ operation of DB is performed.
D: Delete – when user ‘Remove’ any record from the system, ‘Delete’ operation of DB is performed.

It does not matter at all, which DB is used and how the operation is preformed. End user has no concern if any join or sub-query, trigger or stored-procedure, query or function was used to do what he wanted. But, the interesting thing is that all DB operations performed by user, from UI of any application, is one of the above four, acronym as CRUD.

As a database tester one should be focusing on following DB testing activities:

What to test in database testing:

1) Ensure data mapping:

Make sure that the mapping between different forms or screens of AUT and the Relations of its DB is not only accurate but is also according to design documents. For all CRUD operations, verify that respective tables and records are updated when user clicks ‘Save’, ‘Update’, ‘Search’ or ‘Delete’ from GUI of the application.

2) Ensure ACID Properties of Transactions:

ACID properties of DB Transactions refer to the ‘Atomicity’, ‘Consistency’, ‘Isolation’ and ‘Durability’. Proper testing of these four properties must be done during the DB testing activity. This area demands more rigorous, thorough and keen testing when the database is distributed.

3) Ensure Data Integrity:

Consider that different modules (i.e. screens or forms) of application use the same data in different ways and perform all the CRUD operations on the data. In that case, make it sure that the latest state of data is reflected everywhere. System must show the updated and most recent values or the status of such shared data on all the forms and screens. This is called the Data Integrity.

4) Ensure Accuracy of implemented Business Rules:

Today, databases are not meant only to store the records. In fact, DBs have been evolved into extremely powerful tools that provide ample support to the developers in order to implement the business logic at DB level. Some simple examples of powerful features of DBs are ‘Referential Integrity’, relational constrains, triggers and stored procedures. So, using these and many other features offered by DBs, developers implement the business logic on DB level. Tester must ensure that the implemented business logic is correct and works accurately.

Above points describe the four most important ‘What Tos’ of database testing. Now, I will put some light on ‘How Tos’ of DB Testing. But, first of all I feel it better to explicitly mention an important point. DB Testing is a business critical task, and it should never be assigned to a fresh or inexperienced resource without proper training.

How To Test Database:

1. Create your own Queries

In order to test the DB properly and accurately, first of all a tester should have very good knowledge of SQL and specially DML (Data Manipulation Language) statements. Secondly, the tester should acquire good understanding of internal DB structure of AUT. If these two pre-requisites are fulfilled, then the tester is ready to test DB with complete confidence. (S)He will perform any CRUD operation from the UI of application, and will verify the result using SQL query.

This is the best and robust way of DB testing especially for applications with small to medium level of complexity. Yet, the two pre-requisites described are necessary. Otherwise, this way of DB testing cannot be adopted by the tester.

Moreover, if the application is very complex then it may be hard or impossible for the tester to write all of the needed SQL queries himself or herself. However, for some complex queries, tester may get help from the developer too. I always recommend this method for the testers because it does not only give them the confidence on the testing they have performed but, also enhance their SQL skill.

2. Observe data table by table

If the tester is not good in SQL, then he or she may verify the result of CRUD operation, performed using GUI of the application, by viewing the tables (relations) of DB. Yet, this way may be a bit tedious and cumbersome especially when the DB and tables have large amount of data.

Similarly, this way of DB testing may be extremely difficult for tester if the data to be verified belongs to multiple tables. This way of DB testing also requires at least good knowledge of Table structure of AUT.

3. Get query from developer

This is the simplest way for the tester to test the DB. Perform any CRUD operation from GUI and verify its impacts by executing the respective SQL query obtained from the developer. It requires neither good knowledge of SQL nor good knowledge of application’s DB structure.

So, this method seems easy and good choice for testing DB. But, its drawback is havoc. What if the query given by the developer is semantically wrong or does not fulfill the user’s requirement correctly? In this situation, the client will report the issue and will demand its fix as the best case. While, the worst case is that client may refuse to accept the application.

Conclusion:

Database is the core and critical part of almost every software application. So DB testing of an application demands keen attention, good SQL skills, proper knowledge of DB structure of AUT and proper training.

In order to have the confident test report of this activity, this task should be assigned to a resource with all the four qualities stated above. Otherwise, shipment time surprises, bugs identification by the client, improper or unintended application’s behavior or even wrong outputs of business critical tasks are more likely to be observed. Get this task done by most suitable resources and pay it the well-deserved attention.

How to write a good bug report? Tips and Tricks

Why good Bug report?
If your bug report is effective, chances are higher that it will get fixed. So fixing a bug depends on how effectively you report it. Reporting a bug is nothing but a skill and I will tell you how to achieve this skill.

“The point of writing problem report(bug report) is to get bugs fixed” – By Cem Kaner. If tester is not reporting bug correctly, programmer will most likely reject this bug stating as irreproducible. This can hurt testers moral and some time ego also. (I suggest do not keep any type of ego. Ego’s like “I have reported bug correctly”, “I can reproduce it”, “Why he/she has rejected the bug?”, “It’s not my fault” etc etc..)

What are the qualities of a good software bug report?
Anyone can write a bug report. But not everyone can write a effective bug report. You should be able to distinguish between average bug report and a good bug report. How to distinguish a good or bad bug report? It’s simple, apply following characteristics and techniques to report a bug.

1) Having clearly specified bug number:
Always assign a unique number to each bug report. This will help to identify the bug record. If you are using any automated bug-reporting tool then this unique number will be generated automatically each time you report the bug. Note the number and brief description of each bug you reported.

2) Reproducible:
If your bug is not reproducible it will never get fixed. You should clearly mention the steps to reproduce the bug. Do not assume or skip any reproducing step. Step by step described bug problem is easy to reproduce and fix.

3) Be Specific:
Do not write a essay about the problem. Be Specific and to the point. Try to summarize the problem in minimum words yet in effective way. Do not combine multiple problems even they seem to be similar. Write different reports for each problem.

How to Report a Bug?

Use following simple Bug report template:
This is a simple bug report format. It may vary on the bug report tool you are using. If you are writing bug report manually then some fields need to specifically mention like Bug number which should be assigned manually.

Reporter: Your name and email address.

Product: In which product you found this bug.

Version: The product version if any.

Component: These are the major sub modules of the product.

Platform: Mention the hardware platform where you found this bug. The various platforms like ‘PC’, ‘MAC’, ‘HP’, ‘Sun’ etc.

Operating system: Mention all operating systems where you found the bug. Operating systems like Windows, Linux, Unix, SunOS, Mac OS. Mention the different OS versions also if applicable like Windows NT, Windows 2000, Windows XP etc.

Priority:
When bug should be fixed? Priority is generally set from P1 to P5. P1 as “fix the bug with highest priority” and P5 as ” Fix when time permits”.

Severity:
This describes the impact of the bug.
Types of Severity:

• Blocker: No further testing work can be done.



• Critical: Application crash, Loss of data.



• Major: Major loss of function.



• Minor: minor loss of function.



• Trivial: Some UI enhancements.



• Enhancement: Request for new feature or some enhancement in existing one.

Status:
When you are logging the bug in any bug tracking system then by default the bug status is ‘New’.
Later on bug goes through various stages like Fixed, Verified, Reopen, Won’t Fix etc.

Bug life cycle

What is Bug/Defect?

Simple Wikipedia definition of Bug is: “A computer bug is an error, flaw, mistake, failure, or fault in a computer program that prevents it from working correctly or produces an incorrect result. Bugs arise from mistakes and errors, made by people, in either a program’s source code or its design.”

Other definitions can be:
An unwanted and unintended property of a program or piece of hardware, especially one that causes it to malfunction.

or
A fault in a program, which causes the program to perform in an unintended or unanticipated manner.

Lastly the general definition of bug is: “failure to conform to specifications”.

If you want to detect and resolve the defect in early development stage, defect tracking and software development phases should start simultaneously.

We will discuss more on Writing effective bug report in another article. Let’s concentrate here on bug/defect life cycle.

Life cycle of Bug:

1) Log new defect
When tester logs any new bug the mandatory fields are:
Build version, Submit On, Product, Module, Severity, Synopsis and Description to Reproduce

In above list you can add some optional fields if you are using manual Bug submission template:
These Optional Fields are: Customer name, Browser, Operating system, File Attachments or screenshots.

The following fields remain either specified or blank:
If you have authority to add bug Status, Priority and ‘Assigned to’ fields them you can specify these fields. Otherwise Test manager will set status, Bug priority and assign the bug to respective module owner.

The Above figure is quite complicated but when you consider the significant steps in bug life cycle you will get quick idea of bug life.

On successful logging the bug is reviewed by Development or Test manager. Test manager can set the bug status as Open, can Assign the bug to developer or bug may be deferred until next release.

When bug gets assigned to developer and can start working on it. Developer can set bug status as won’t fix, Couldn’t reproduce, Need more information or ‘Fixed’.

If the bug status set by developer is either ‘Need more info’ or Fixed then QA responds with specific action. If bug is fixed then QA verifies the bug and can set the bug status as verified closed or Reopen.

Bug status description:
These are various stages of bug life cycle. The status caption may vary depending on the bug tracking system you are using.

1) New: When QA files new bug.

2) Deferred: If the bug is not related to current build or can not be fixed in this release or bug is not important to fix immediately then the project manager can set the bug status as deferred.

3) Assigned: ‘Assigned to’ field is set by project lead or manager and assigns bug to developer.

4) Resolved/Fixed: When developer makes necessary code changes and verifies the changes then he/she can make bug status as ‘Fixed’ and the bug is passed to testing team.

5) Could not reproduce: If developer is not able to reproduce the bug by the steps given in bug report by QA then developer can mark the bug as ‘CNR’. QA needs action to check if bug is reproduced and can assign to developer with detailed reproducing steps.

6) Need more information: If developer is not clear about the bug reproduce steps provided by QA to reproduce the bug, then he/she can mark it as “Need more information’. In this case QA needs to add detailed reproducing steps and assign bug back to dev for fix.

7) Reopen: If QA is not satisfy with the fix and if bug is still reproducible even after fix then QA can mark it as ‘Reopen’ so that developer can take appropriate action.

8 ) Closed: If bug is verified by the QA team and if the fix is ok and problem is solved then QA can mark bug as ‘Closed’.

9) Rejected/Invalid: Some times developer or team lead can mark the bug as Rejected or invalid if the system is working according to specifications and bug is just due to some misinterpretation.

Software Testing Advice for Novice

Novice testers have many questions about software testing and the actual work that they are going to perform. As novice testers, you should be aware of certain facts in the software testing profession. The tips below will certainly help to advance you in your software-testing career. These ‘testing truths’ are applicable to and helpful for experienced testing professionals as well. Apply each and every testing truth mentioned below in your career and you will never regret what you do.

Know Your Application
Don’t start testing without understanding the requirements. If you test without knowledge of the requirements, you will not be able to determine if a program is functioning as designed and you will not be able to tell if required functionality is missing. Clear knowledge of requirements, before starting testing, is a must for any tester.

Know Your Domain
As I have said many times, you should acquire a thorough knowledge of the domain on which you are working. Knowing the domain will help you suggest good bug solutions. Your test manager will appreciate your suggestions, if you have valid points to make. Don’t stop by only logging the bug. Provide solutions as well. Good domain knowledge will also help you to design better test cases with maximum test coverage.

No Assumptions In Testing
Don’t start testing with the assumption that there will be no errors. As a tester, you should always be looking for errors.

Learn New Technologies
No doubt, old testing techniques still play a vital role in day-to-day testing, but try to introduce new testing procedures that work for you. Don’t rely on book knowledge. Be practical. Your new testing ideas may work amazingly for you.

You Can’t Guarantee a Bug Free Application
No matter how much testing you perform, you can’t guarantee a 100% bug free application. There are some constraints that may force your team to advance a product to the next level, knowing some common or low priority issues remain. Try to explore as many bugs as you can, but prioritize your efforts on basic and crucial functions. Put your best efforts doing good work.

Think Like An End User
This is my top piece of advice. Don’t think only like a technical guy. Think like customers or end users. Also, always think beyond your end users. Test your application as an end user. Think how an end user will be using your application. Technical plus end user thinking will assure that your application is user friendly and will pass acceptance tests easily. This was the first advice to me from my test manager when I was a novice tester.

100% Test Coverage Is Not Possible
Don’t obsess about 100% test coverage. There are millions of inputs and test combinations that are simply impossible to cover. Use techniques like boundary value analysis and equivalence partitioning testing to limit your test cases to manageable sizes.

Build Good Relations With Developers
As a tester, you communicate with many other team members, especially developers. There are many situations where tester and developer may not agree on certain points. It will take your skill to handle such situations without harming a good relationship with the developer. If you are wrong, admit it. If you are right, be diplomatic. Don’t take it personally. After all, it is a profession, and you both want a good product.

Learn From Mistakes
As a novice, you will make mistakes. If you don’t make mistakes, you are not testing hard enough! You will learn things as you get experience. Use these mistakes as your learning experience. Try not to repeat the same mistakes. It hurts when the client files any bug in an application tested by you. It is definitely an embracing situation for you and cannot be avoided. However, don’t beat yourself up. Find the root cause of the failure. Try to find out why you didn’t find that bug, and avoid the same mistake in the future. If required, change some testing procedures you are following.

Don’t Underestimate Yourself if Some of Your bugs Are Not Fixed
Some testers have assumptions that all bugs logged by them should get fixed. It is a good point to a certain level but you must be flexible according to the situation. All bugs may or may not be fixed. Management can defer bugs to fix later as some bugs have low priority, low severity or no time to fix. Over time you will also learn which bugs can be deferred until the next release.

Over To You:
If you are an experienced tester, what advice do you like to give to novice testers?

What if there isn’t enough time for thorough testing

Sometimes Tester need common sense to test a application!!!

I am saying this because most of the times it is not possible to test the whole application within the specified time. In such situations it’s better to find out the risk factors in the projects and concentrate on them.

Here are some points to be considered when you are in such a situation:
1) Find out Important functionality is your project?
2) Find out High-risk module of the project?
3) Which functionality is most visible to the user?
4) Which functionality has the largest safety impact?
5) Which functionality has the largest financial impact on users?
6) Which aspects of the application are most important to the customer?
7) Which parts of the code are most complex, and thus most subject to errors?
8) Which parts of the application were developed in rush or panic mode?
9) What do the developers think are the highest-risk aspects of the application?
10) What kinds of problems would cause the worst publicity?
11) What kinds of problems would cause the most customer service complaints?
12) What kinds of tests could easily cover multiple functionalities?

Considering these points you can greatly reduce the risk of project releasing under less time constraint.