Introduction to Software Testing
Rob Pirozzi, LogiGear Corporation
This article will provide a brief introduction and overview of software testing, discuss both manual and automated testing, and introduce the concept of blended testing. It is intended as a brief primer on the software testing process.
Software testing is both a discipline and a process. Though software testing is part of the software development process, it should not be considered part of software development. It is a separate discipline from software development. Software development is the process of coding functionality to meet defined end-user needs. Software testing is an iterative process of both validating functionality, and, even more important, attempting to break the software. The iterative process of software testing consists of:
1、 Designing tests
2、 Executing tests
3、 Identifying problems
4、 Getting problems fixed
The objective of software testing is to find problems and fix them to improve quality. Software testing typically represents 40% of a software development budget.
While Software testing tends to be considered a part of development, it is really its own discipline and should be tracked as its own project. Software testing, while working very closely with development, should be independent enough to be able to hold-up or slow product delivery if quality objectives are not met.
Types of Software Testing
Software testing consists of several subcategories, each of which is done for different purposes, and often using different techniques. Software testing categories include:
1、 Functionality testing to verify the proper functionality of the software, including validation of system and business requirements, validation of formulas and calculations, as well as testing of user interface functionality.
2、 Forced error testing, or attempting to break and fix the software during testing so that customers do not break it in production. 2、Forced error testing, or attempting to break and fix the software during testing so that customers do not break it in production.vvv
3、 Compatibility testing to ensure that software is compatible with various hardware platforms, operating systems, other software packages, and even previous releases of the same software.
4、 Performance testing to see how well software performs in terms of the speed of computations and responsiveness to the end-user.
5、 Scalability testing to ensure that the software will function well as the number of users and size of databases increase.
6、 Stress testing to see how the system performs under extreme conditions, such as a very large number of simultaneous users.
7、 Usability testing to ensure that the software is easy and intuitive to use.
8、 Application security testing to make sure that valuable and sensitive data cannot be accessed inappropriately or compromised under concerted attack.
In some cases, there may even have to be other types of testing such as regulatory-compliance testing, depending on the type of software and intended industry.
Methods of Software Testing
There are two basic methods of performing software testing:
1、Manual testing
2、Automated testing
Manual Software Testing
As the name would imply, manual software testing is the process of an individual or individuals manually testing software. This can take the form of navigating user interfaces, submitting information, or even trying to hack the software or underlying database. As one might presume, manual software testing is labor-intensive and slow. There are some things for which manual software testing is appropriate, including:
1、 User interface or usability testing
2、 Exploratory/ad hoc testing (where testers do not follow a ‘script’, but rather testers ‘explore’ the application and use their instincts to find bugs)
3、 Testing areas of the application which experience a lot of change.
4、 User acceptance testing (often, this can also be automated)
The time commitment involved with manual software testing is one of its most significant drawbacks. The time needed to fully test the system will typically range from weeks to months. Variability of results depending on who is performing the tests can also be a problem. For these reasons, many companies look to automation as a means of accelerating the software testing process while minimizing the variability of results.
Automated Software Testing
Automated software testing is the process of creating test scripts that can then be run automatically, repetitively, and through many iterations. Done properly, automated software testing can help to minimize the variability of results, speed up the testing process, increase test coverage (the number of different things tested), and ultimately provide greater confidence in the quality of the software being tested.
There are, however, some things for which automated software testing is not appropriate. These include:
1、 End user usability testing is not typically a good candidate for automated testing.
2、 Tests which will not be run more than a couple of times are typically not a good candidate for automated tasting, since the payoff of in test automation comes after many test executions.
3、 Tests for areas of the application which experience a lot of change are also not a good candidate for automation since this can lead to substantial maintenance of test automation scripts. Such areas of the application may be more effectively tested manually.
It is important to note that test automation is software, and just like the software you are building for internal or external customers, it must be well-architected. A good test automation architecture, such as a keyword-driven testing framework, will reduce the overall cost of ownership of your test automation by minimizing maintenance expense and increasing the number of automated tests, allowing your organization to run more tests (and achieve higher quality) for the same investment of time and money.
Achieving the Right Blend of Software Testing
An effective software testing process is typically a mix of test types, executed through a combination of manual and automated testing. The mix and number of tests is determined by the quality requirements of the application (i.e. Is it mission critical? Is time-to-market the most important factor?). Each method (automated or manual) is used for what is appropriate.
Manual testing is best leveraged for those tests which require spontaneity and creativity, as well a good deal of subjectivity:
1、 User interface or usability testing
2、 Exploratory/ad hoc testing
While automated testing is best used for tests which are explicit and repetitive:
1、General QA and functionality tests (i.e. does each module do what the requirements say it should? How does the application respond to incorrect inputs?)
2、‘End to end’ scenario tests (simulating a ‘real world’ use of the software in a production environment)
3、Performance, load, and stress testing
In practice, 80 to 95% of tests should be able to be automated. It is typically not the tests which limit the amount of automation, but rather a limit in the number or ability of testing staff, or technical limitations in the test automation tool to interact with the application. The right combination of tests, technology, and people can lead to a very high level of test automation and all the inherent benefits.
Conclusion
As one can see, software testing is a very large undertaking. It is very important to have an effective process in place prior to embarking on any major software development effort. This is essential to the success of the overall effort, and delivering quality software to end-users.
Things I Tell New Testers
By Randall W. Rice, CSQA, CST, CSTM
As I have led and trained test teams over the years, there are some things that I routinely make sure I convey to new testers. Whether you are new to testing, or a seasoned professional, these are helpful things to remember.
1. You are an inspector – You can’t guarantee quality.
Many testers get sidetracked by not understanding that they assess things, not control them. There is a huge difference between the two! For example, a team may work weeks on a test and find many defects, only to see management decide to release the product with known serious defects. The team often feels demoralized and asks why they even do testing.
I ask the team members if they got paid, and most of the time the answer is “yes.” I ask if they did their jobs to the best of their ability, and once again, most of the time they answer “yes.” I then tell them, “You did your job. You tested to the best of your ability, found defects and reported them. Now, go home and relax. In fact, the only way you can fail as a tester is to fail to report a known defect.”
This may not improve morale much, but it does help keep things in perspective, especially in having the ability to not take the daily frustrations of work home every night.
Many testers, me included, when we first start out in testing, seem to feel personally responsible for the quality of the application or system we are testing. While the work ethic is admirable, we testers really have little control of the product quality. It is for this reason that testers do not ensure quality. The problem is that management doesn’t always see that distinction. This is seen when management asks questions like, “Aren’t we paying these people to have good software?”
2. Defects are valuable.
Each defect is an opportunity to learn and improve. We may only get one opportunity to observe a defect, so I always tell testers to be observant and not fall prey to the boredom of testing.
Defect information is perhaps one of the most telling sources of project data available. However, that depends on how well we capture and convey accurate information about the defects we find.
Each defect costs the organization money. If we don’t learn from them, we are wasting a lot of time and money. The leverage happens when we convert a mistake to a learning opportunity. Let’s face it – some lessons are only learned by experience.
It doesn’t do any good to place blame over a defect. All blame does is lower morale and shut down communication. It’s like beating the proverbial dead horse, hoping it will come back to life!
I wrote an article called “Defects are Good” which explores in more depth about the positive side of defects.
3. Everything is fine until you report the first problem.
This is where the reality of being a tester sinks in. You can plan the test, get all of the needed resources in place and it seems everyone is on your side. However, things tend to get tenser when you report that first problem.
The reason for the sudden change in attitude is that now you are criticizing someone’s work. There is a pride of ownership that causes egos to be bruised and relationships to be strained. Some of this is to be expected, just be aware that attitudes may change when you start finding problems.
One thing I always suggest to testers is to read some of your past defect reports as if you were the recipient. You may find that you need to be more diplomatic. (See how gently I said that?) It may not be as much fun to write a defect report without sarcasm, but it will help maintain good relationships.
4. You can only test what you can observe.
You may want to test that really creative case, but if you don’t have a way to observe the results, what’s the point? Although some applications allow you to observe a lot, there are still things you may not have access to, such as structural views, hidden objects, background processing, etc.
5.Never forget how you get someplace.
I’m not speaking about knowing why you walked into a room, but of the steps you took in testing something. It’s common for beginning testers to find a significant defect, only to fail to recreate it for resolution. Then, you’re left with the uncomfortable feeling of not knowing if you truly found a defect, or just used the application incorrectly.
Some ways you can trace your steps include test scripts, a test log, keystroke loggers such as Spector (http://www.spectorsoft.com/) and screen video capture tools such as Hypercam (http://www.hyperionics.com/)
6.Standards and processes are your friend.
Although the idea of standards and processes may seem confining to some, they give you valuable guidance in doing your job. Don’t reject standards because they may be detailed and lengthy. Instead, use them as a guide to do your job faster and more consistently.
7.There’s never enough time for testing.
Almost every tester complains there is not enough time for testing, but the reality is there is never enough time to test anything to a degree of completeness. This is especially true when you consider software attributes, such as usability, security, compatibility, interoperability, etc.
Instead of complaining about the lack of time, learn to prioritize based on risk and focus on the application objectives important to senior management. Sometimes we test more than we need to because our objectives are out of line with what the sponsors value.
8. You can’t find all of the defects.
Don’t get discouraged if a defect is found later in something you tested. You may do a very complete job and achieve a high level of defect removal, but 100% is an elusive goal.
9.Maintain a sense of humor and perspective
Being able to laugh and keep a healthy sense of balance may be your best survival mechanism. If you are in a bad situation, remember, this too will pass.
10. Strive for excellence instead of perfection
New testers often get caught up in the quest for perfection with the attitude that 100% correctness is the standard. I was a victim of this as well, but in my defense, I was influenced heavily by the TQM posters of the late 80’s and articles such as “99.9% Isn’t Good Enough.”
The problem with perfectionism is that it will make your progress slower, introduce fear into everything you do, cause you to become more critical of others, and generally, drive even your friends and family to the point of despair.
Of course, no one wants mistakes, but they happen regardless. To think otherwise is to deny reality. Excellence is a habit that is shown in your attitude and dedication to the work you do. If you are committed to be the best, you will go the extra mile.
My observation is that most people are forgiving when they see a mistake or experience a lapse in service. The thing people pay the most attention to is your response to the problem.
11. Developers are not the enemy
It takes the entire project team to deliver a quality project. While it may seem at times that the developers don’t care about quality, there may be more issues than meet the eye. You will make more progress by working with developers instead of against them. Remember that good communication is a major key to successful projects (and testing). When you take an adversarial position, the communication stops and so does much of your information needed for testing.
12. Build and maintain a personal network
Your personal and professional relationships are a great asset. They are a support system that can help when you have a job and when you don’t. Find a mentor and when you learn enough lessons, become a mentor to someone else.
13.Keep building your skills
Your skills are what distinguish you from others. Always keep learning by attending professional meetings, obtaining certifications, and reading in the profession. I make it a goal to read at least one book a week on topics related to by personal and professional growth (testing, leadership, business, IT, etc.).
One personal development guru noted that if you read 30 minutes a day on any given topic, in five years you’ll be an expert on the topic. It worked for me – try it for yourself!
Another great way to stay sharp and to build a network is to stay active in some of the QA and testing forums. Some of my favorites are at http://www.stickyminds.com/, http://www.qaforums.com/, and to sound our own bell, we are starting forums at http://www.riceconsulting.com/.
14. When the going gets tough, the lazy get creative
I made a sign with this saying on it and hung it on my desk when I first became a test team leader. It reminded me to let creativity be my point of leverage in problem solving.
Learn to see problems in a new and creative way. You may have a good test plan, but how will you deal with changes? Flexibility is a key attribute of good problem-solving leaders.
15.Simple is not always easy.
Much of what we do in testing seems simple. However, the challenge is in the consistency required to maintain the effort.
Some solutions to problems may seem too simple at first, but don’t discard an idea just because is it simple and obvious. Also, don’t underestimate the effort needed to implement a simple idea!
Some readers of the book I co-wrote with William E. Perry, Surviving the Top Ten Challenges of Software Testing, have commented that the challenges are simple and easy to solve. That makes me wonder why people still raise the same “people problems” year after year. I think it is easier to obtain the head knowledge than to actually implement the solutions at work.
Conclusion
Before knowledge, comes wisdom. You may learn a lot of testing techniques, but your ability to apply them will be limited if you don’t have the wisdom to know when to apply them and the context to understand them. One of the problems you have in starting out in just about anything is that "you don't know what you don't know." Wisdom helps you understand what you need to know to be successful.
The things I have listed are things I wish I had fully realized when I first started testing. I hope they serve you well.
