Software Testing Approaches And Techniques

Introduction

We know the difference between Unit Testing and Integration Testing. But sometimes, we may mix the Unit Test with Integration Testing. In this article mainly I'll focus on white box testing. If we know the the unit testing principles and best practice technique, then it will be easy to maintain the quality of the application. It is also most importaint to know, how to write the test cases using Behavior Driven Development (BDD) during white-box testing.

Topics

White-box & black-box testing
Advantage and disadvantages of unit testing
Principles to write unit testing
Best practice technique to write unit testing
Writing test scenarios, test case & test data
Traditional unit testing principle
Naming convention of the test method
Why we need Behavior Driven Development (BDD)
Avoid tight coupling object
Unit test object mocking

Basic overview of the software testing

Benefit of software testing

Reduce risk of failures when systems are transferred to production or live operation.
Documented proof that business requirements have been met.
Assurance that users are able to operate designed solution productively.
Assurance that the system works properly with existing legacy systems.

Test focusing point between developers and QA testers

Developer's point of view

Developers verify the implemented logics or codes of the modules of the application according to the business requirements. They make sure that, all the logics can run smoothly. On the other hand, they confirm that, if end-user passes any unexpected data then the logics still can handle the exception as well as it can show the proper messages. This is known as white-box testing because they can see their internal logics or codes. For example, automated unit testing or automated integration testing.

QA tester's point of view

Quality assurance (QA) testers verify the modules of the application according to the business requirements. But they don't need to worry about the logics or codes. They test the functionality as an end-user. They input the positive data to verify the positive functionalities. On the other hand, they put the wrong data to make sure that they get the proper messages for the wrong input. This is known as a black box testing because, they can't see the internal logics or codes of the modules of the application. They verify the external behaviors only. For example, acceptance testing.

Let's drill down the basic testing concept

Test documentation basic

Test case - A set of conditions and steps to determine whether a system satisfies the requirements correctly or not. This is described by test data, environment and expected result.
Test suite - a collection of test cases.
Test plan - document describing testing approach, test suites and test cases.
Test strategy - a way to identify test cases from the specification.
Test effectiveness - relative ability of test strategy to find bugs.
Test coverage - the percentage of testable elements that have been tested.

Test scenarios & test cases

If you have to withdraw money from an ATM machine, then it is a scenario. But to withdraw money, you need to execute some actions (test cases).

Test scenarios - A test-scenarios have multiple test cases. So while starting testing, first prepare test scenarios, then create test cases for each scenario. Example, Checking the functionality of the login button.
Test case - A test case is a condition which is executed for expected output with predefined set of steps with known inputs. Examples, Test-Case1: Click on the button without entering user-name and password; Test-Case2: Click on the button only entering user-name; Test-Case3: Click on the button while entering wrong user-name and wrong password. etc. These all the test-cases will have some expected, unexpected and exceptional results.

Term of mistake

Error - mistake made by developers during implementation of a software system.
Failure - incorrect behavior of a program.
Fault - incorrect code that caused a failure.
Incident - symptoms associated with a failure.
Bug - an error or fault.
Fault directed testing – finding bugs through failure.

Positive & negative testing

Positive testing: When tester tests the application with valid input/data from positive point of view then it is known as positive testing.
Negative testing: When tester tests the application with invalid input/data from negative point of view then it is known as negative testing.
Exceptional testing: When tester tests the application with data and get an exceptional error then it is known as exceptional testing.

White box & black box testing

Depending on the developers and QA testers, we can say that we have two types of testing. Developers can test the internal logics of the implemented modules of the application system or development box. This box is visible to the developers and it's called white-box.
Other the hand, the QA testers can't see the internal logics of the source codes and they can see only the front-end functionalities of the system as an end user. That's why, the box is black and they test the external behaviors. It is called black-box testing.

Advantages & disadvantages of white box testing

Advantages - Reveals bugs in hidden code; Forces reason in implementation for positive, negative and exceptional cases; Get confident and documental proof; Improving design and code quality.
Disadvantages - Expensive comparing to time and design; Need proper knowledge about testing methodology; Cases missed can miss out code.

Advantages & disadvantages of black box testing

Advantages: Tester and programmer are independent; Testing from end user’s point of view; Test cases can be prepared right after specification are decided.
Disadvantages: Duplication of test cases by testers and developers; Only simple testing can be done as all cases can’t be build and tested.

White box testing vs. black box approach

White box approach: Statement coverage; Decision coverage; Condition coverage; Path coverage.
Black box approach: Field level check; Field level validation; User interface check; Function level check.

Type of testing

Unit testing; Integration testing; Functional testing; System testing; Stress testing; Performance testing; Usability testing; Acceptance testing; Regression testing; Beta testing.

Most common testing in many organizations

Most common testing in many organizations

Granularity levels

Granularity levels in Testing

Unit test - Verification of single classes.
Module test - Testing interaction of groups of classes (package).
Integration test - Verification of interactions between modules/components.
Functional test - Verification of external behaviors of modules, components and system.
System test - Testing of the system against objectives.
Acceptance test - Validation of application against user requirements.
Regression test - Re-running all tests on system when it is changed.

Speared the unit test

Top 5 Excuses For Not Doing Unit Testing

I don’t have time to do the unit test.
The office pays me to write down the codes, not to write down unit tests.
I am supporting a legacy application without unit tests and existing design are not suitable for a unit test.
QA and user acceptance testing are far more effective in finding bugs.
I don’t know how to write unit tests.

Advantages of unit test

Reduces the level of bugs in production code; Saves your development time; Automated tests can be run as frequently as required; Make it easier to change and refactor code; Improve the design of code especially with Test-Driven-Development; A form of documentation; Inspires confidence!; Measures of completion.

Disadvantages of unit test

Not implementable to logical operators; Insensitive to loop operators (number of iterations).

Best practice & principles to write unit test

Principles to write unit test

First principle, “Test the logic of the class only, nothing else”

One of the most important principles during unit testing: when you are going to test a class, you should not have dependency on database, file, registry, web services, etc. You should be able to test any class in complete "isolation" and it should be designed to support complete "isolation." According to this principle mock out all external services and state and remember unit test NEVER uses- configuration settings; a database; Another Application/Service/file/network I/O; logging.

Scond principle, “Fail first and set a guard at the door"

Before you write piece of logic, first write your fail test. After that, add or refactor your guard logics and throw proper exceptions or messages for the negative or exceptional data. Finally run it and pass the test.

Explanation of the unit test principles

First principle

Suppose, you have a method 'IsValidUser' to verify the user name and password for login.

Best practice to write unit test

Now if you look at the line no. 15 and 17, then you will see class 'UserLogIn' has a dependency to the class 'UserDataAccess' and it is tightly coupled. Now the problem is that, you need to test a method 'IsValidUser' of a class 'UserLogIn' and you have to avoid the dependency. Because, if you call 'IsValidUser', then it will call the method 'GetUserInfoByUseName'. According to the unit test principle, if you have a dependency from one method to another out sider class, then you have to mock object for that outsider class. It means that you have to inject some dummy data into the outsider classs-object. So in this example, you need to inject dummy data for the 'GetUserInfoByUseeName(userName). If you do so, then you will able to verify the logics of the method 'IsValidUser' in the 'UserLogIn' class.

So, avoid the tight coupling code and refactor your the code again. Find my other article to know dependency, tight coupling and loose coupling. Now my main goal to make you know the unit testing principle only. Anyway, let's refactor the class.

Best practice to write unit test

Now the class 'RefactoringUserLogIn' is loosely coupled and you can inject the implemented class of the 'IUserDataAccess' interface which will make your life easy as far as unit testing. Note that, there are many ways to inject the implemented class. I'll explain later how to mock an object of a class. During unit testing, if you mock the 'IUserDataAccess' it means you are implementing a dummy object of 'IUserDataAccess' interface and the method 'IsValidUser' will not able to get any data directly from outsider like the database or ORM via 'UserDataAccess' class.

Scond Principle

Say, the first time more than 50% people fail their road driving test, THEN "practicing on the highway"; "seeking tips from others"; - they ensure the success in the road test.

Anyway, If you're already familiar with Test-Driven-Development (TDD), then you're already familiar about the fail test. I will explain TDD also bellow in this article.

Let’s explain the guard logics and how it will help you to pass your unit testing. Say, you have a method ‘GetSumByPositiveNumber’ and it has two parameters. The business requirement is always input the positive numbers and get return the sum. Never input a negative or zero number.

Best practice to write unit test

Now according to your business requirements, if you pass the values of parameters like 1 and 2 then the method will return 3 which is correct. But what happens if you pass the negative values as a parameter or some combination like (-5, -1), (-5, 1), (5, 0), (0, 0) etc.?

But if you put some guard logics into the method at the beginning then the test should be passed. Because you may throw the exception for the negative data.

Best practice to write unit test

Best practices for writing unit test

If old test fails for the extension of the method which is introduced, then remove or enhance old test and avoid requirement conflict.
If you remove a single line in your tested class and all tests still pass, then you haven’t enough unit tests.
Developer will often run the test, so make test easy to run.
Test only public methods and public interface of the component
Don't create instances of classes directly inside a unit test - use factory method.
Refactor the unit test and "Keep It Simple". Remember that tests must be maintained by any code - so better to keep it simple.
Any test should run in any order; so avoid dependencies between tests.
Test should be readable like a book; so write comments in Asserts. Write descriptive method names. Use Behavior-Driven-Development (BBD) Technique
Test everything that could possibly break.
Test everything, but not Private Methods.

Writing test scenario, test cases & test data

Only three test cases for unit testing, nothing else

Positive Test Cases

Correct data to check for correct output;

Negative Test Cases

Broken or missing data to check for proper handling;

Exception Test Cases

Giving unexpected data or behavior and check for the exception caught properly or not.

Test data according to the test cases

Let’s set some test data and consider “GetSum()” as an example. Here my main goal to show you, how you can set some TEST DATA for the test cases.

Best practice to write unit test

Type of Test Data: Positive data; Negative data; Exceptional data.

Positive data for positive test cases

The main goal of the positive test cases, to verify the functionality of the logics.

TestCase-1: Given positive values, should return expected result; TestData-1: Set input parameters as firstNumber =1, secondNumber=1.

Negative data for negative test cases

The main goal of the negative test cases, to get proper messages for the bad input according to your business requirements.

TestCase-2: Given invalid values, should produce invalid argument message

TestData-2: Set Input Parameters as firstNumber =-1, secondNumber =-1

TestData-3: Set Input Parameters as firstNumber =-1, secondNumber = 1

TestData-4: Set Input Parameters as firstNumber = 0, secondNumber = 1

TestData-5: Set Input Parameters as firstNumber = 0, secondNumber =-1

TestData-6: Set Input Parameters as firstNumber = 0, secondNumber = 0 etc.

Exceptional data for exceptional test cases

The main goal of the exceptional test cases, to find out the proper exception handling with proper messages. So that, your code can’t break for the threshold limits. Here you can set the threshold limits of your test data. In DOT NET, the minimum value for a variable of type int is -2147483648 and the maximum value for a variable of type int is 2147483647.

TestCase3: Given threshold limit values, should throw exception message

TestData-7: Set Input Parameters as firstNumber =1, secondNumber =2147483649

TestData-8: Set Input Parameters as firstNumber =1, secondNumber = -2147483649

TestData-9: Set Input Parameters as firstNumber =2147483649 secondNumber = -2147483649

TestData-10: Set Input Parameters as firstNumber =2147483647, secondNumber=2147483647 etc.

Naming convention of the test method for unit testing

Traditional principle of unit test

One test method is written to test one and only one method and one assert method should test only one expectation at a time. In a short the principle says – “one function/method and one assert per test method”. So, let’s consider the bellow example,

Best practice to write unit test

Comparing the traditional principle to the real world

Test scenario, verify the “GetSum” method. Test cases,

Positive test cases

TC1: Given positive values, should return expected result

Test Data-1: firstValue =5, secondValue =6

Negative test cases

TC2: Given zero values, should produce invalid argument message

Test Data-2: firstValue =0, secondValue =0

TC3: Given negative values, should produce invalid argument message

Test Data-3: firstValue =-5, secondValue =-6

Exceptional test cases

TC4: Given threshold limit values, should throw exception message

Test Data-4: firstValue =2147483647, secondValue =2147483647.

Test method example: Now according to the traditional principle, let’s write the test method for the “GetSum”

Best practice to write unit test

Now according to the traditional principle we have covered the positive test Case with “Test Data-1”. But what about negative and exceptional test cases?? How do we cover the negative and exceptional test cases with traditional principle??

Behavior Driven Development (BDD)

Why need BDD

If you want to cover all the behaviors of your test cases according to your previous example then you need to follow some technique; so that, you can write down all the behaviors of the method. So, The BDD is the technique which gives you the opportunity to fulfill all of the test cases with standard and readable naming convention. "Many peoples, many minds", there are many techniques to write the naming convention of the test method. But it really depends on you and your preference. If you follow some other technique, then there is nothing right or wrong directly; because, everything is relative. Anyway, in short you can say that, "in BDD, component tests the expected behavior".

Concept of BDD

Given: I am a beginner to the BDD technique, and I never use this technique before
When: I read this tutorial for BDD
Then: I have started to like it and finally I learn it.

BDD naming convention

Test Scenario: verify the “GetSum” method.

Test Cases

Positive Test Cases: TC1: Given positive values, should return expected result; Test data-1: firstValue =5, secondValue =6.

Test method & naming convection

GivenPositiveVaidValuesAsParams_WhenGetSumIsCalled_ThenItShouldReturnSumValue

Body structure of the test method

There is no hard code rules for that. We generally follow the structure because it’s easy to read and understand. The general AAA structure are given bellow for the Test Method.

AAA structure

Arrange
Act
Assert

Arrange

In this arrange portion, you need to declare variables and you have to create the instance of the objects of the classes.

AAA structure

Act

The verified method is called in the portion. This is mainly used for passing the input parameters into the method and collecting the return result from the calling method.

AAA structure

Assert

In this portion we compare the actual result of the calling method with the expected result. The test is passed or failed depends on this assert portion.

AAA structure

Object mocking for unit testing

Why do we need mock

Suppose, you need to test the behavior of one method and it has an external service or method within it. During the unit testing, you can avoid this kind of external dependencies and the mock technique which gives you the facility to avoid the behavioral test of the external method or service. Let’s see the example,

Object mocking for unit testing

This GetSum method has no external dependencies within it. So, it does not need object mocking.

Now see the bellow example of the class and this has two dependencies; that are CheckingAccount and SavingAccount.

Object mocking for unit testing

Now you may want to verify the method “GetTotalMoneyByUserAccount” and it has two dependencies within it. So, to solve this kind of situation, you need mocking.

Mock framework

There are many mock frameworks like Typemock Isolator, Rhino Mocks, Moq and NMock. You can use any one of them.

Requirement before making mock

The class can’t be sealed; The method can’t be static; but if you need then use adaptor pattern; you can mock if Interfaces, Abstract methods or properties or Virtual Methods or properties on concrete classes.

Adding MOQ into the unit testing project: Select your Unit Testing Project. Go to Reference> Click Right button of the mouse and select “Manage NuGet Packages”. Now write down ‘Moq’ into the search textbox and finally install it to your project.

Mock framework

Object mocking using MOQ

Object mocking using MOQ

In the above example line number 35 and 41, first we are creating instances of the object using Mock.

Mock<IBankAccount> mockCheckingAccount = new Mock<IBankAccount>();
Mock<IBankAccount> mockSavingAccount = new Mock<IBankAccount>();

In the line numbers 38 and 44, you are implementing the dummy implementation and if the "GetMoneyByUserAccountNo" method is called then it will return the 5 and 6. So you don't need to know any internal logics of that.

mockCheckingAccount.Setup(a => a.GetMoneyByUserAccountNo(userAccountNumber)).Returns(5);
mockSavingAccount.Setup(a => a.GetMoneyByUserAccountNo(userAccountNumber)).Returns(6);

Now line number 48 shows that you are injecting the mock objects into the class via constructor.

SimpleMath simpleMath = new SimpleMath(mockCheckingAccount.Object, mockSavingAccount.Object);

Secondly in the line number 56 of the 'Act' portion, you are calling the verified method and result is into the actualResult variable and finally inline number 59, which is giving you the actual result.

actualSumResult = simpleMath.GetTotalMoneyByUserAccount(userAccountNumber);

My main goal here, is to introduce you to the object mocking world. So this is just an example, you can use any of the mocking frameworks. NOTE that, I didn’t care about the latest version of the packages; because, this is just an example and I use my old images and codes. So, you should use the latest version of the packages.

How to call order of test methods using moq sequences

Download Moq.Sequences.dll from github then add Moq.Sequences.dll as a reference in your .Net project and add a using Moq.Sequences; in your test class. Or simply use "Manage NuGet Packages” Manager and type "Moq.Sequences" and search it.

Finally install it to your project and use it to call as a reference.

How to call order of test methods using moq sequences

Moq.Sequences supports the following

Checks order of the method calls, property gets and property sets; Allows you to specify the number of times a call is made before the next one is expected; Allows inter-mixing of sequenced and non-sequenced expectations; Thread safe – each thread can have its own sequence

Example for calling methods sequentially

[TestMethod]  
public void Given_Valid_Data_For_Object_Model_When_TestMethod_Is_Called_Then_It_Should_Be_Done_Successfully()  
{  
    ......  
    ......  
  
    //// Methods are called sequentially.  
    using (Sequence.Create())  
    {  
        //// First, it is called  
        Given_Valid_Template_Data_When_MethodSave_Is_Called_Then_It_Should_Be_Saved_Successfully();  
  
         //// Second, it is called.  
        Given_Valid_Template_Value_When_MethodUpdate_Is_Called_Then_It_Should_Update_Template();  
  
        ///// Third, it is called.  
        Given_Valid_Template_ID_When_DeleteTemplate_Is_Called_Then_It_Should_Delete_Template();  
    }  
  
    ......  
    ......  
}  
[TestMethod]  
public void Given_Valid_Data_For_Model_When_TestMethod_Is_Called_Then_It_Should_Be_Done_Successfully()  
{  
    ......  
    ......  
    ///  
    using (Sequence.Create())  
    {  
        //// Methods are called sequentially.  
        mockObject.Setup(x => x.MethodA()).InSequence();  
        mockObject.Setup(x=>x.MethodB()).InSequence(Times.AtMostOnce());  
        mockObject.Setup(x=> x.MethodC()).InSequence(Times.Once());  
    }  
    ......  
    ......  
}

Regression testing

How we introduce new bugs

Software enhancement and maintenance are now our daily work. New addition or modification of the existing components or modules or any error correction can sometimes introduce new bugs. So in short, I can say that when I fix one bug then maybe, I'm introducing new bugs.

What's our expectation?

Everybody knows the answer and that is, we want continuous improvement. But this new enhancement should not affect any existing tested modules or features.

Where is the solution

Yes, you are right it is the Regression Testing which is the solution of this problem. Because it verifies that current bugs are fixed and newly added functionality has not affected any other part of the application.

What exactly we do in regression testing

We rerun the previous tests and verify the current result to our previous expected results.

Regression testing is not re-testing

Regression Testing ensures that new changes are not affecting the other part of the application; Re-Testing means testing bugs or the functionality again to ensure that the code fixes the bug. If it is not fixed then we need to reopen the defect. Otherwise if defect is fixed, then close the defect.