# 2026 - A Testing Philosophy > Author: Thomas Kottke > Published: 2026-7-10 > Canonical: https://tdkottke.com/blog/002-testing/ I was recently asked by an interviewer to describe my testing philosophy. I stumbled through my answer but I could tell it was unsatisfactory. Around the same time I also came across the same challenge at work. In that context, I was the leader attempting guide my team to a more robust testing strategy. As a result of that I produced a document AND was able to codify the strategy into an LLM persona. Below you will find my current testing philosophy and the approach I take to testing software applications. I hope you find it useful. --- As a software engineer it is crucial to my success that the applications I write work. Tests inherently improve the quality of code because it requires that you can break down the code into small composable chunks and verify the functionality of each chunk/unit. This document outlines my current testing philosophy and how I approach testing for _traditional software application_. ==Note: AI Testing (aka "evaluations") is a different challenge.== > [!tldr] > I believe that testing adds value through validation and regression checks to any piece of software and anything "production ready" should be well tested. > > The tests themselves should follow these principles: > 1. Tests only give you value when you put value in > - Garbage in/Garbage out > - Descriptions should highlight what is being tested and what value it adds > - Failures should provide meaningful feedback on _why_ it failed > 2. Tests (like code) should be **cumulative** > - Many focused tests are better than one cumulative test > 3. Tests should focus on **behavior** and not implementation > - Answer the question: "_How we expect the code to behave?_" > 4. Tests should be deterministic > - Tests should produce the same result every time ## When not to test (aka Testing Blacklist) In general, adding tests to your code is a net benefit. It helps catch bugs related to information management and operational flow. Below we talk about the types of tests that I feel should be written for an application (see below), I want to first address that just because testing improves quality, does not mean everything needs to be tested. The term "_Testing Blacklist_" refers to the idea that we would _opt out_ of testing specific chunks of code. My preferred way to approach selecting which tests to blacklist is by answering the following question > [!question] Questions > 1. Does the codebase I am testing have control over the outcomes of my tests? > 2. Does the file/code do any processing? These questions will primarily identify things like: 1. Libraries and static setup configurations 2. Configuration files (such as constants, interfaces, or definitions) 3. Registration or Barrel Files (think `index.js` or `__init__.py` files) which typically act as collectors and unified interfaces Everything else, should be tested. ## Testing Types I found that having a clear definition of what a test was and what value it added, provided clarity for both myself and teams that I worked with around what is expected from a test AND what value a test provides. ### Manual Testing (Discouraged) There will be times where the value proposition of writing an automated test does not exist OR that you need a completely different process to test the code than you would need to execute the code. This is where you may still need **manual tests**. This type of test needs to be manually executed by a human. In these scenarios the human doing the testing will need record what they did and how they did it. This allows us to record the value add and determinism that we get from automated tests. ```markdown --- date: // Datetime test was run result: // Pass | Fail tester: // Name of the human doing the testing versions: // Map of packages/libraries being tested // - iEngineerHubServer@1.4.0 // - iEngineer --- # Manual Testing Report | | | | --- | --- | | Test Cases | 1 | | Tests Passing | 1 | | Tests Failing | 0 | ## Test Summary This document outlines the manual testing needed to ## Environment **Base URL:** https://application.example.com **Environment:** Staging | Feature Flag | Enabled | | --- | --- | | ShowDebugging | No | ## Detailed Test Cases ## 01 - User can login **Tests Data:** - Username: user@example.com - Password: ******* **Tests Steps:** 1. Navigate to the application landing page 2. Click the "Sign In" button 3. Login using the users credentials 4. Confirm user home page is shown **Expected Results:** - [ ] Authentication Successful - [ ] Redirected to correctly to the home page - [ ] No Console Errors: [ ] Yes [ ] No **Notes & Issues Encountered:** - ``` The above template can be generated for a manual tester to follow. It is both a guide and an interactive form. The manual tester will configure the environment, follow the steps and fill out the **Expected Results** and **Notes & Issues Encountered** sections for each test. Then populate the test case pass/fail count. ### Developer Testing _Developer Testing_ is a form of testing done during development. These tests are meant to be run in isolation, focusing on logic written in the project. Developer tests should be executable anywhere without the need for externally running dependencies. This allows developers to run these tests locally OR in CI environments without heavy setup The question we want to answer with these types of tests is, "do the individual elements of the application function as we expect them too." This question is broken out into three types of tests to highlight the 3 types of elements we have in our application: _Unit Tests_, _Orchestration Tests_, _External Orchestration Tests_ **Frameworks:** - Javascript/Typescript - Mocha + Chai **General Best Practices** - **Test are written alongside the code** - Test files are written adjacent to the code that is being tested - **Tests should be used to accept new changes** #### Unit Tests Unit tests are the core type of developer test. These will be the most frequently written test and should focus on isolated units of code that use only platform-native dependencies. Examples would include, using the JS `Math` API or the NodeJS `fs` api. These tests should be exhaustive, testing as many scenarios as possible to ensure that there is no cap in how the application will behave to possible inputs. **Best Practices:** - **Unit tests should be unique**: Each test should testa specific scenario or edge case - **Use descriptive test names:** Each test should have a descriptive name which clearly outline what is being tested. Example: `should return error when input is null` is better than `should call validateInput function` because it highlights what the input is and what the expected output is - **Test both positive and negative scenarios:** Write test that cover both they "happy path" (positive tests) as well as unexpected inputs to ensure the unit of code behaves correctly in all situations - **Keep test simple:** Avoid complex logic or conditionals in your tests. If a test is hard to understand, it is hard to maintain #### Orchestration Tests As we write more complex code (and follow the law of composability) we will start to drift away from individual/functional units of code by composing those units together into larger functions. We call these _Orchestration functions_. The real value in this type of test is that we want to isolate the logic the function to coordinate the nested unit functions and NOT the unit functions themselves An example of an _Orchestration function_ might be a function which calls our units based on a condition: ```ts import { add, subtract } from './math.js'; enum Operator { ADD = 'add', SUBTRACT = 'substract', } function doMath(a: number, b: number, operator: Operator): number { switch(operator) { case Operator.ADD: { return add(a, b) } case Operator.SUBTRACT: { return subtract(a, b) } default: throw new Error('Invalid Operator') } } ``` In the above example we have a method called `doMath` which in turn calls an `add` and `subtract` functions. In our Orchestration tests we want to avoid calling add and subtract (since they already have unit tests) and instead focus on making sure that: 1. The correct function (add/subtract) was called based on the operator 2. The function that is called (add/subtract) was passed the correct arguments in the correct order 3. That our system correctly handles invalid operators This ensures we do not duplicate testing the `add` and `subtract` methods. **Best Practices:** - **Focus on the interactions between units** - The primary goal of orchestration tests is to verify the correct units are called based on the inputs provided. Focus on testing the interactions and not the results of the underlying units. - **Use spying when possible over mocking** #### External Orchestration Tests There are rarely cases where an application is 100% self-contained. We call out to Databases, External APIs, Caches, and the File System. All of which are other pieces of software. _Importantly_ those external resources should have already tested themselves (or we should not use them if they are untested). What our tests do need to cover is _verification of data contracts_. When we call a database, we want to verify that we are providing the correct input AND that if the database throws an error, the system can handle the response. **Best Practices:** - **Always mock external services** - We should never make real calls to external services. The input should be spied on and the response should be mocked to produce a deterministic result - **Test all documented error scenarios** - External Orchestration tests should ensure the code can handle failures from those external systems - **Verify input construction with spies** - Use spies to verify that the external orchestration correctly constructs the input arguments. - **Mock realistic response structures** - Use actual response structures from documentation to build real examples. This allows the code to also act as a form of documetnation - **Document the API version** - Keep track of which version of an API or SDK you are testing against so that it is clear if tests are outdated compared to dependencies. ### End-to-end Testing End-to-End (E2E) tests are named such because they test the entire application from start to finish. This can be simulating real scenarios or verifying a multi-step process works as expected. These tests interact with a live environment which uses external services like databases. This ensures the application works in real-world scenarios. The other big variant from Developer Tests is that these should focus on expected paths rather than being exhaustive. E2E tests can further be broken up into 2 sub-categories: _Functional Tests_ and _User Workflow Tests_. **Frameworks:** - Browser Based and REST APIs - Playwright #### Functional Tests *Functional tests* focus on non-user facing processes. These could be background tasks, automated jobs, or system integrations. An example might be a webhook which is triggered by an external service. This may generate some data and it will be important for the system to be able to confirm the data is created correctly. **Best Practices:** - **Always clean up test data** - Leave the environment better than you found it. Delete data after it was created in a test, or revert if it was modified. - **Always use the same data for each run** - Ensure tests are deterministic by reusing the test pattern - **Verify actual state changes** - Make sure to check beyond the UI to make sure changes were applied. Verify in the database, queues or other systems directly - **Test with realistic data** - Use data that resembles production scenarios #### User Workflow Tests As the name implies, _User Workflow Tests_ emulate the users workflow in an application. This is especially common with UI and focuses on a specific _user task_ that needs to be completed, verifying that a user could complete those steps. Each test should focus on a specific _action_ and have an _expected outcome_. Ex: when a user uploads a file, the application should notify them that the file was accepted. From a developers perspective you can think of these tests similar to the testing you do while developing a feature, simply turned into a codified (and repeatable process). **Best Practices:** - **Use semantic selectors**: Prefer selectors based on user-visible attributes like roles (`getByRole('button')`), labels, or discrete test IDs over CSS Selectors like class name or element hierarchy. - **Test from the user's perspective**: Focus on what the users can see and do, not on what is implemented. Test workflows like "user can submit a form" not "form submit handler called" - **Write clear action descriptions**: Describe actions in plain language as if instructing a human tester. Ex: "Click the Submit Button" rather than "Trigger click event on \#submit-btn" - **Clean up test data**: Remove any data that was created after the test has been completed (pass or fail) - **Use meaningful waits**: Wait for specific conditions (element visible, text appears) rather than arbitrary timeouts. This ensures tests fail faster AND it is more clear _why_ the test failed. ## Testing Anti-Patterns Understanding what _not to do_ is just as important as knowing what to do. Here are common mistakes that make test brittle, unreliable, ineffective ### Testing Log Messages Logs are a common component of our applications. They let us record snapshots at points in time to see what is happening or when something happened. ```ts ❌ DONT test('should create a record', () => { const consoleSpy = jest.spyOn(console, 'log'); createRecord({}) expect(consoleSpy).toHavebeenCalledWith('Record with ID: X - Created') }) ✅ DO test('should create a record', () => { }) ``` **Why is this a problem?** - Log messages are nondeterministic, change frequently, and can be changed independent of the actual functionality. ### Appendix A: Playwright Test Documentation When writing playwright tests, I have found a couple of helpful patterns that allow me to organize tests as well as describe their value. One of the challenges with most testing frameworks is capturing intent behind a test. Playwright gives us [annotations](https://playwright.dev/docs/test-annotations) as a tool to attach metadata about the test. I have found that I can use this to my advantage and have constructed this interface for a `TestSuite`: ```ts interface TestSuite { id: number; name: string; notes: string; tags: []; description: string; // Description of what the suite is testing purpose: string; // Describes the value being added by the feature users?: []; // List of user personas to use for testing steps: Array<{ // List of descrete steps to be executed tags: []; action: string; // Describes what action is being taken expectedOutcome: string // Describes what we expect to happen test: Function // The actutal test logic }> } ``` This can be combined with Playwright to create a comprehensive report that not only shows the test results but also describes the why and how of the test. ### Appendix B: Playwright Tags Most test runners offer a way to "grep" for specific tests or to run a sub-set of tests in a single execution. One way playwright does this is through _tags_. What I want to highlight in this section is not that tags are important but that tags should be a way to collect tests. I want to highlight here a couple of examples of tags which I have identified as useful for testing as well as where I use those tests. > [!note] Notation - Developer Tests > For Developer Tests, I typically append the tags to the end of the test name: > > ```ts > // Mocha > > enum TestTypes { > UNIT = '[unit]', > ORCHESTRATION = '[orchestration]', > EXTERNAL = '[external-orchestration]' > } > > it(`\should create a new record ${TestTypes.UNIT}\`, () => {}) > > it(`should throw an error when the input is invalid ${TestTypes.UNIT}\`, () => {}) > ``` > [!note] Notation - E2E Testing > For E2E Tests, I use the playwright Tag System: > > ```ts > // Playwright > > enum TestTypes { > FUNCTIONAL = '@functional', > USER_WORKFLOE = '@user-workflow', > SMOKE = '@smoke', > COMPREHENSIVE = '@comprehensive' > } > > it(`\should create a new record ${TestTypes.SMOKE}\`, () => {}) > > it(`should throw an error when the input is invalid ${TestTypes.COMPREHENSIVE}\`, () => {}) > ``` Some common types of tests that I like to have are: | Type | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Test Types | These are the types of test described above in [Test Types](#test_types) | | Smoke | Smoke tests are tests which can be run quickly to verify core functionality. These should be fast so they can be run frequently against the system. This may be case where mocking apis would be acceptable | | Comprehensive | Unlike smoke tests, a comprehensive test should be detailed. It should take the time to ensure things run until completion. They verify complete feature functionality. They should be used to verify everything is working after a deployment | | | |