By mcaaltuntas


2009-02-06 12:08:34 8 Comments

What is Unit test, Integration Test, Smoke test, Regression Test and what are the differences between them? And Which tools can I use for each of them?

For example I use JUnit and NUnit for Unit testing and Integration Testing. Are there any Smoke Test or Regression Test tools?

21 comments

@Yuri Adeodato 2019-02-05 19:04:35

In a simplified way.

Unit test: Test a single piece of code, algorithm, class or system. This test should be independent and the dependencies should be mocked or stubbed.

Integration test: should test if the component, algorithm, class or system work well when used by other components, Integration tests is not for testing how system work (behavior), it should test if the system works well.

Smoke test: Is a very small set of test that should run first them a large set of tests, it will just ensure that the most critical features of the system works even after a upgrade.

Regression test: Is the process of testing changes to computer programs to make sure that the older programming still works with the new changes. Is a bigger set of tests than smoke tests.

If you want to know more about integration test you can go into this course in Udemy, it has a good discount.

https://www.udemy.com/testes-de-integracao-com-spring-boot/?couponCode=TISB_ODESC2019

@令狐葱 2019-01-18 09:53:59

Unit tests Unit tests are very low level, close to the source of your application. They consist in testing individual methods and functions of the classes, components or modules used by your software. Unit tests are in general quite cheap to automate and can be run very quickly by a continuous integration server.

Integration tests Integration tests verify that different modules or services used by your application work well together. For example, it can be testing the interaction with the database or making sure that microservices work together as expected. These types of tests are more expensive to run as they require multiple parts of the application to be up and running.

Functional tests Functional tests focus on the business requirements of an application. They only verify the output of an action and do not check the intermediate states of the system when performing that action.

There is sometimes a confusion between integration tests and functional tests as they both require multiple components to interact with each other. The difference is that an integration test may simply verify that you can query the database while a functional test would expect to get a specific value from the database as defined by the product requirements.

End-to-end tests End-to-end testing replicates a user behavior with the software in a complete application environment. It verifies that various user flows work as expected and can be as simple as loading a web page or logging in or much more complex scenarios verifying email notifications, online payments, etc...

End-to-end tests are very useful, but they're expensive to perform and can be hard to maintain when they're automated. It is recommended to have a few key end-to-end tests and rely more on lower level types of testing (unit and integration tests) to be able to quickly identify breaking changes.

Acceptance testing Acceptance tests are formal tests executed to verify if a system satisfies its business requirements. They require the entire application to be up and running and focus on replicating user behaviors. But they can also go further and measure the performance of the system and reject changes if certain goals are not met.

Performance testing Performance tests check the behaviors of the system when it is under significant load. These tests are non-functional and can have the various form to understand the reliability, stability, and availability of the platform. For instance, it can be observing response times when executing a high number of requests, or seeing how the system behaves with a significant of data.

Performance tests are by their nature quite costly to implement and run, but they can help you understand if new changes are going to degrade your system.

Smoke testing Smoke tests are basic tests that check basic functionality of the application. They are meant to be quick to execute, and their goal is to give you the assurance that the major features of your system are working as expected.

Smoke tests can be useful right after a new build is made to decide whether or not you can run more expensive tests, or right after a deployment to make sure that they application is running properly in the newly deployed environment.

source: https://www.atlassian.com/continuous-delivery/software-testing/types-of-software-testing

@yerlilbilgin 2019-05-20 07:14:44

The definition of Smoke testing here is quite poor. Any low level test can be a 'basic test that check basic functionality of the application'. The best candidate for this definition is unit tests. A unit test tests a unit of the application (as the name implies) and a unit is indeed a basic funtionality (depending on the definition of functionality). The best definition seems to be the one provided by @ddaa

@ddaa 2009-02-06 12:28:48

  • Unit test: Specify and test one point of the contract of single method of a class. This should have a very narrow and well defined scope. Complex dependencies and interactions to the outside world are stubbed or mocked.

  • Integration test: Test the correct inter-operation of multiple subsystems. There is whole spectrum there, from testing integration between two classes, to testing integration with the production environment.

  • Smoke test (aka Sanity check): A simple integration test where we just check that when the system under test is invoked it returns normally and does not blow up.

    • Smoke testing is both an analogy with electronics, where the first test occurs when powering up a circuit (if it smokes, it's bad!)...
    • ... and, apparently, with plumbing, where a system of pipes is literally filled by smoke and then checked visually. If anything smokes, the system is leaky.
  • Regression test: A test that was written when a bug was fixed. It ensures that this specific bug will not occur again. The full name is "non-regression test". It can also be a test made prior to changing an application to make sure the application provides the same outcome.

To this, I will add:

  • Acceptance test: Test that a feature or use case is correctly implemented. It is similar to an integration test, but with a focus on the use case to provide rather than on the components involved.

  • System test: Tests a system as a black box. Dependencies on other systems are often mocked or stubbed during the test (otherwise it would be more of an integration test).

  • Pre-flight check: Tests that are repeated in a production-like environment, to alleviate the 'builds on my machine' syndrome. Often this is realized by doing an acceptance or smoke test in a production like environment.

@SnakE 2012-05-10 12:52:57

Smoke testing predates electronics by a century and comes from plumbing, when a system of pipes were filled by an actual smoke and then checked visually. If it smoked, it was leaky.

@BobRodes 2012-12-13 19:53:13

Regression tests are also used for feature changes, not just bug fixes. Nikita's answer below is a more comprehensive definition.

@Cristi Diaconescu 2013-06-05 11:58:35

@AndyM The background of 'Smoke test' is inaccurate. IIRC it comes from plumbing, where smoke is pumped in the system of pipes after it's built (and before it's connected to the water supply). If any smoke comes out, the pipes are not properly sealed. This is less damaging than actually letting the water flow and see if any puddles occur (possibly damaging walls/masonry in the process). It's a gross approximation that the system will not fail catastrophically. A dev process may be: "Build " passed? => "Smoke test" passed ? => "Acceptance test" passed => out to QA team for detailed testing.

@Ben 2013-09-30 00:51:16

@CristiDiaconescu - you impressed me anyway. Better get on TV before Trebek retires.

@Aykut Kllic 2013-12-20 00:11:11

Are there any simple sample code that shows all those types?

@jjpcondor 2014-04-06 05:26:51

Acceptance testing is a testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria of a given use case. It is usually performed by the expert-user, customer or other authorized entity to determine whether or not the system is acceptable. Test Pilot evaluation and certification. A test pilot is an aviator who tests new aircraft by fling specific maneuvers,

@Brian C 2016-06-23 17:56:16

I believe you made an error in stating that a "Regression Test" really shorthand for "Non-Regression Test"? I am skeptical, in part because that's just unintuitive and confusing (though there are plenty of terms that are), but also Wikipedia has two separate articles on Regression and Non-Regression Tests. The article on Regression testing even says: Contrast with non-regression testing... which aims to verify whether, after introducing or updating a given software application, the change has had the intended effect.

@ddaa 2016-06-27 11:19:23

I do not think that this distinction is useful. One just wants regressions not to pop up. Tests written before a regression happens are just tests. Regression tests are significant because bugs that actually happened tend to come back unless tested.

@Bharat 2016-12-02 06:43:45

@ddaa Sanity testing and smoke testing are not same. Sanity testing is performed after the build has clear the Smoke test and has been accepted by QA team for further testing, sanity testing checks the major functionality with finer details.

@rolls 2017-04-05 00:48:00

They still perform smoke tests on pressurised intake tracts of vehicles. It is quite common.

@vokimon 2018-07-18 19:09:50

«It can also be a test made prior to changing an application to make sure the application provides the same outcome.» This is usually called a back-to-back test. Just ensure that old and new versions provide the same result for a given input. Not the ones you want to rely on but often the only way to get test control on some untested or closed software.

@Matthew Madson 2018-12-28 21:16:09

It might be worth discussing the difference between solitary and sociable unit tests. See: martinfowler.com/bliki/UnitTest.html See: youtube.com/watch?v=EZ05e7EMOLM

@Monads are like... 2019-01-23 13:43:11

Another one is end-to-end tests: These are automated tests that run on your real production environment and try to imitate a real user behaviour and will usually include a few core flows of your product.

@Saurabh Oza 2019-03-10 03:57:00

you can find the unit testing and mockito insights in below link : onlyfullstack.blogspot.com/2019/02/junit-tutorial.html and onlyfullstack.blogspot.com/2019/02/mockito-tutorial.html

@phyzome 2019-04-28 11:30:20

On regression tests, I suspect that the meaning of the term has changed over time. Your definition of "A test that was written when a bug was fixed" is exactly how it is used in my corner of industry, but I think there was an older definition of "do the features still work as expected" that is now just called "testing".

@Blue Clouds 2017-12-09 08:45:56

Smoke test has been explained here already and is simple. Regression test comes under integration test.

Automated tests can be divided into just 2.

Unit test and Integration Test. (this is all that matters)

I would call use the phrase "long test"(LT) for all tests like integration test, functional test, regression test, UI test , etc. And unit test as "short test".

A LT example could be, automatically loading a web page, logging in to the account and buying a book. If the test passes it is more likely to run on live site the same way(hence the 'better sleep' reference). Long = distance between web page(start) and database(end).

And this is a great article discussing the benefits of integration testing(long test) over unit testing

@Radostta 2017-11-17 11:58:04

Smoke and sanity testing are both performed after a software build to identify whether to start testing. Sanity may or may not be executed after smoke testing. They can be executed separately or at the same time - sanity being immediately after smoke.

Because sanity testing is more in-depth and takes more time, in most cases it is well worthed to be automated.

Smoke testing usually takes no longer than 5-30 minutes for execution. It is more general: it checks a small number of core functionalities of the whole system, in order to verify that the stability of the software is good enough for further testing and that there are no issues, blocking the run of the planned test cases.

Sanity testing is more detailed than smoke and may take from 15 minutes up to a whole day, depending on the scale of the new build. It is a more specialized type of acceptance testing, performed after progression or re-testing. It checks the core features of certain new functionalities and/or bug fixes together with some closely related to them features, in order to verify that they are functioning as to the required operational logic, before regression testing can be executed in larger scale.

@Kaleem Ullah 2015-11-26 07:28:31

Answer from one of the best Websites for Software Testing Techniques:

Types of Software Testing – Complete List Click Here

enter image description here

It's quite a long description, I'm not going to paste it here: but it may be helpful for someone who wants to know all the testing techniques.

Hope it'll be helpful :)

@Krunal 2017-07-18 12:29:27

Unit Testing: It always performs by developer after their development done to find out issue from their testing side before they make any requirement ready for QA.

Integration Testing: It means tester have to verify module to sub module verification when some data/function output are drive to one module to other module. Or in your system if using third party tool which using your system data for integrate.

Smoke Testing: tester performed to verify system for high-level testing and trying to find out show stopper bug before changes or code goes live.

Regression Testing: Tester performed regression for verification of existing functionality due to changes implemented in system for newly enhancement or changes in system.

@Vin Shahrdar 2017-11-29 15:55:00

Don't we have to create the test before doing the actual development?

@Krunal 2017-12-01 10:24:03

@VinShahrdar , Are you talking about Unit testing ?

@Vin Shahrdar 2017-12-02 22:10:18

yes. I usually create my unit tests before writing production code. That's how you're supposed to do it, correct?

@Krunal 2017-12-06 11:52:14

Yeah .. But unit testing also performs before QA faced by the developer. Before deploying code on QA server dev always perform unit testing

@Sanyal 2015-07-26 04:23:41

Regression test - Is a type of SW testing where we try to cover or check around the bug Fix. the functionality around the bug fix should not get changed or altered due to the Fix provided. Issues found in such process are called as Regression Issues.

Smoke Testing : Is a kind of testing done to decide whether to accept the Build/Software for further QA testing .

@Dave 2014-10-12 19:55:36

Unit testing is directed at the smallest part of the implementation possible. In java this means you are testing a single class. If the class depends on other classes these are faked.

When your test calls more than one class, its an integration test.

Full test suites can take a long time to run, so after a change many teams run some quick to complete tests to detect significant breakages. For example, you have broken the URIs to essential resources. These are the smoke tests.

Regression tests run on every build and allow you to refactor effectively by catching what you break. Any kind of test can be regression test, but I find unit tests are most helpful finding the source of fault.

@Madhivanan 2014-06-30 09:50:43

  • Integration Testing : Integration Testing is the Integrate the Another Element
  • Smoke Testing : Smoke Testing is also known as build version Testing.Smoke testing is the initial testing process exercised to check whether the software under test is ready/stable for further testing.
  • Regression Testing : Regression Testing is Reapeated Testing. Whether new software is effected in another module or not.
  • Unit Testing : It is a White box Testing .Only Developers involve in it

@mohit sarsar 2014-09-16 10:50:07

Unit Testing:- unit testing is usually done by the developers side,where as testers are partly evolved in this type of testing where testing is done unit by unit. In java Junit test cases can also be possible to test whether the written code is perfectly designed or not.

Integration Testing:- This type of testing is possible after the unit testing when all/some components are integrated.This type of testing will make sure that when components are integrated,do they affect each others working capabilities or functionalists.

Smoke Testing:- This type of testing is done at the last when system is integrated successfully and ready to go on production server. This type of testing will make sure that every important functionality from start to end is working fine and system is ready to deploy on production server.

Regression Testing:- This type of testing is important to test that unintended/unwanted defects are not present in the system when developer fixed some issues. This testing also make sure that all the bugs are successfully solved and because of that no other issues are occurred.

@malini 2014-04-26 12:02:23

unit test: testing of individual module or independent component in an application is known to be unit testing , the unit testing will be done by developer.

integration test: combining all the modules and testing the application to verify the communication and the data flow between the modules are working properly or not , this testing also performed by developers.

smoke test IN smoke test they check the application in shallow and wide manner, In smoke testing they check the main functionality of the application , if there is any blocker issue in the application they will report to developer team , and developing team will fix it and rectify the defect, and give it back to testing team and now testing team will check all the modules to verify tat changes made in one module will impact the other module or not. IN SMOKE TESTING the test cases are scripted

regression testing executing the same test cases repeatedly to ensure tat the unchanged module does not cause any defect. REGRESSION TESTING comes under functional testing

@Jaime Gago 2013-10-28 21:19:35

One type of test that seems to be worth mentioning in this thread is stress/performance/load tests which could be simply put as finding out the limits beyond which a certain piece of software breaks. Note that in terms of tooling it is essential to precisely determine the scope of what one proposes to stress tests from a system perspective. For instance in the case of a "web application" stress testing can include in its scope the web server application itself and so the tooling could intervene on that end. Here is a nice post about http load testing

@CharlesB 2013-10-30 06:46:48

You're right, but you don't really answer the question...

@Jaime Gago 2013-10-31 07:54:06

I agree but I thought it would be a useful extension to the question just as the comment from @AndyM about Canary testing. If there is a more opened thread where all software testing types are to be listed and defined I haven't found it.

@nhfghfgsq 2013-09-17 09:59:06

Unit test: Verifying that particular component (i.e.,class) created or modified functions as designed. This test can be manual or automated but does not move beyond the boundary of the component.

Integration test: Verifying that the interaction of particular components function as designed. Integration tests can be performed at the unit level or the system level. These tests can be manual or automated.

Regression test: Verifying that new defects are not introduced into existing code. These tests can be manual or automated.

Depending upon your SDLC (waterfall, rup, agile, etc) particular tests may be performed in 'phases' or may all be performed, more or less, at the same time. For example, unit testing may be limited to developers who then turn the code over to testers for integration and regression testing. However another approach might have developers doing unit testing and some level of integration and regression testing (using a TDD approach along with continuous integration and automated unit and regression tests).

@AndyM 2013-09-11 03:47:51

A new test category I've just become aware of is the:

Canary test

A Canary test is an automated, non-destructive test that is run on a regular basis in a LIVE environment, such that if it ever fails, something really bad has happened.

Examples might be:

  • Has data that should only ever be available in DEV/TEST appeared in LIVE.
  • Has a background process failed to run
  • Can a user logon

@Dan Dascalescu 2014-02-23 23:53:46

Can the site be pinged at all - suitably enough, a service called Binary Canary exists.

@Robino 2014-09-03 12:06:48

The name comes from coal mining: take canary with you "down t'pit". When it snuffs it, get out quick. Canaries are very sensitive to small concentrations of noxious gas and would die before the concentrations levels became toxic to humans. If a Canary test fails, fix it quickly because it'll only be a matter of time before LIVE fails.

@00prometheus 2016-09-15 19:23:42

The way we use Canary testing at my job is to first shift a few customers over to a new version, rather than all of them at once. If the first few customers survive, we can add the rest. Those first few are the canaries.

@GregNash 2017-01-19 22:32:54

@00prometheus, that's beta testing.

@Harvey Lin 2017-01-25 18:25:49

At my work we use TestComplete to run automatic tests on our product, especially apps with GUIs, does that counts as Canary test?

@Charles Roberto Canato 2017-07-19 00:00:25

@HarveyLin, although a Canary test is necessarily a test that prevents disaster, of course it's not only used this way. But the rule of thumb is "test this is working 'cause it IS critical". Of course, every test has almost this same goal, but the concept is very specific. In your case, I wouldn't count all of the tests as Canary ones.

@Charles Roberto Canato 2017-07-19 00:01:59

@00prometheus and GregNash, that's more specifically a Canary Release.

@Nikita G 2010-10-21 05:28:26

REGRESSION TESTING-

"A regression test re-runs previous tests against the changed software to ensure that the changes made in the current software do not affect the functionality of the existing software."

@Daryl 2013-03-22 21:26:16

Where are you quoting from?

@Zach Lysobey 2013-10-29 15:58:26

According to this page, that quote came from the Wikipedia "Software testing" article though it seems that the passage has been changed at some point since 2010.

@Rainald62 2018-07-24 15:14:36

Anyways, WP is not a valid source. Sources referenced there may be valid. There are no valid sources referenced in WP, neither in the articles nor on the discussion pages, that would support the claim that the "non-" makes any difference. I compared the text snippets in the result lists from searches in Google Books for both "regression test" and "non-regression test". It's the same.

@rdrex 2009-03-04 01:48:41

Unit test: Verifying that particular component (i.e.,class) created or modified functions as designed. This test can be manual or automated but does not move beyond the boundary of the component.

Integration test: Verifying that the interaction of particular components function as designed. Integration tests can be performed at the unit level or the system level. These tests can be manual or automated.

Regression test: Verifying that new defects are not introduced into existing code. These tests can be manual or automated.

Depending upon your SDLC (waterfall, rup, agile, etc) particular tests may be performed in 'phases' or may all be performed, more or less, at the same time. For example, unit testing may be limited to developers who then turn the code over to testers for integration and regression testing. However another approach might have developers doing unit testing and some level of integration and regression testing (using a TDD approach along with continuous integration and automated unit and regression tests).

The tool set will depend largely on the codebase but there are many open source tools for unit testing (JUnit). HP's (mercury) QTP or Borland's Silktest are both tools for automated integration and regression testing.

@uriDium 2009-02-06 12:31:31

Some good answers already but I would like further refine them:

Unit testing is the only form of white box testing here. The others are black box testing. White box testing means that you know the input, you know the inner workings of the mechanism and can inspect it and you know the output. With black box testing you only know what the input is and what the output should be.

So clearly unit testing is the only white box testing here.

  • Unit testing test specific pieces of code. Usually methods.
  • Integration testing test whether your new feature piece of software can intergrate with everything else.
  • Regression testing. This is testing done to make sure you haven't broken anything. Everything that used to work should still work.
  • Smoke testing is done as a quick test to make sure everything looks okay before you get involved in the more vigorous testing.

@joel.neely 2009-02-06 12:51:51

Unit testing is not necessarily white-box. Some of the best unit tests I've seen are essentially examples drawn from the requirements, specifying expected results independently from any implementation concepts.

@Lieven Keersmaekers 2009-02-06 13:29:27

added to that, your unit tests are included in your regression tests therefore regression tests are neither white or black box tests. I'd go as far to say that even integration and smoke tests can be either white- or blackbox tests.

@Hazok 2012-12-13 05:25:26

I would have to disagree with this. Testing a design pattern implementation is a form of integration testing and is white box testing.

@annakata 2009-02-06 12:27:23

apocryphal historical trivia: "smoke testing" comes from submarine engineering (inherited from plumbing) where literal smoke would be pumped into the hull to see if any of it came out again, which would be rather a dramatic failure for a submarine!

@Dan Dascalescu 2014-02-23 23:52:59

@Jon Skeet 2009-02-06 12:15:07

Everyone will have slightly different definitions, and there are often grey areas. However:

  • Unit test: does this one little bit (as isolated as possible) work?
  • Integration test: do these two (or more) components work together?
  • Smoke test: does this whole system (as close to being a production system as possible) hang together reasonably well? (i.e. are we reasonably confident it won't create a black hole?)
  • Regression test: have we inadvertently re-introduced any bugs we'd previously fixed?

@alpha_989 2018-02-15 23:38:02

How do you place your integration tests with respect to unit-tests? If myprj is the main project directory, and mypkg is located under myprj, I have the unit tests located under myprj/tests/test_module1.py, and my package located under myprj/mypkg. This works great for unit tests, but I wonder if there is any convention, I should follow for where the integration tests should reside?

@Jon Skeet 2018-02-16 03:25:29

@alpha_989: I don't know what the convention would be for Python. In .NET I currently have the production code, unit tests and integration tests in three separate projects, peers of each other - but there are lots of alternatives too.

@alpha_989 2018-02-16 04:04:22

Ok.. thanks. I could find standard recommendation other than looking at another python project. but I will follow yours..

@Gerrie Schenck 2009-02-06 12:13:56

  • Unit test: an automatic test to test the internal workings of a class. It should be a stand-alone test which is not related to other resources.
  • Integration test: an automatic test that is done on an environment, so similar to unit tests but with external resources (db, disk access)
  • Regression test: after implementing new features or bug fixes, you re-test scenarios which worked in the past. Here you cover the possibility in which your new features break existing features.
  • Smoke testing: first tests on which testers can conclude if they will continue testing.

@Robert Koritnik 2010-10-11 14:37:01

Regression test's definition isn't really exactly how it is. @ddaa defines it correctly.

@rogerdpack 2015-08-04 16:31:48

The definition of Integration Test is definitely fuzzy. For instance on the answer here stackoverflow.com/a/4904533/32453 it's more defined as testing multiple interactions of your code, not necessarily needing a real DB (external resource)...though some people define it the way you've described...ahh terminology. (I do somewhat prefer the former definition, FWIW, testing multiple interactions.)

Related Questions

Sponsored Content

48 Answered Questions

32 Answered Questions

[SOLVED] What exactly is RESTful programming?

  • 2009-03-22 14:45:39
  • hasen
  • 1599655 View
  • 3839 Score
  • 32 Answer
  • Tags:   http rest definition

10 Answered Questions

[SOLVED] What's the difference between faking, mocking, and stubbing?

19 Answered Questions

[SOLVED] What is the difference between integration and unit tests?

31 Answered Questions

[SOLVED] Unit Testing C Code

5 Answered Questions

8 Answered Questions

[SOLVED] What does “DAMP not DRY” mean when talking about unit tests?

  • 2011-06-23 11:22:49
  • Ian Ringrose
  • 53959 View
  • 298 Score
  • 8 Answer
  • Tags:   unit-testing

12 Answered Questions

[SOLVED] Unit test naming best practices

8 Answered Questions

13 Answered Questions

[SOLVED] Unit tests vs Functional tests

Sponsored Content