|
> I think the test name separation by unit test/integration test/micro
test/behaviour test doesn't work, but I'm not sure what's the "sensible"
way to separate them yet like fast/slow/io/non-io? business/technical?structure vs behaviour?
I'm curious about this, because I hear it from time to time.
I have some questions, and we don't have to limit those to Tony, although I'm also interested in his replies:
1. What kind of "doesn't work"? It works for me, so maybe we have different ideas about how it could work or should work.
2. I classify tests in order to better understand all the different kinds of intent and purpose we have when we write them. This helps me decide how to choose which tests to write next. What challenges do you have with all these test classifications?
3. Some people report that there are too many test classifications to understand well. They become confused. I empathize. Why don't you simply ignore those classifications until you need them?
Finally, as for the difference between business and technical tests, when I talk about TDD I tend to focus on technical tests, because that's my context for TDD: to focus on the code. I handle Customer Tests (or business tests) quite differently, and I only sometimes practise what we've called Story-TDD or Acceptance-TDD. I practise it most often when I play the role of Customer for myself, such as on my volunteer projects. I try _very hard_ to clarify this for the people I teach, but I always run the risk that the message doesn't make it through. -- J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
-- J. B. (Joe) Rainsberger :: :: :: Teaching evolutionary design and TDD since 2002
|
I agree that the naming can be confusing because often the same name means different things to different people. I don't get too hung up on the naming of types of tests (though I love Gpaw's "microtests" because it gets out of the "unit test" mire). Instead, I tray to talk about the meaning the other person has behind the name.
When I started doing TDD, I sorted my tests into three categories:
- "unit tests" which tested in memory code without any other dependencies
- "database tests" which tested code dependent on the database. This led me to using the Adapter Pattern so I could isolate my unit tests from the database and test only the adapter against a real database.
- "deployed tests" which required the system to be deployed in order to run. These tended to be "story tests," though I found that by delegating from the requirements of the app server (J2EE in those days) to Plain Old Java Objects with the same API, I could implement most of the story tests the same way as unit tests, so "story tests" became another category.
Eventually I had need to call other systems beyond the database, so those became another classification of tests, but done in the same fashion as the database tests.
- George
toggle quoted message
Show quoted text
On 7/4/23 9:38 AM, J. B. Rainsberger wrote: > I think the test name separation by unit test/integration test/micro
test/behaviour test doesn't work, but I'm not sure what's the "sensible" way to separate them yetlike fast/slow/io/non-io? business/technical?structure vs behaviour?
I'm curious about this, because I hear it from time to time.
I have some questions, and we don't have to limit those to Tony, although I'm also interested in his replies:
1. What kind of "doesn't work"? It works for me, so maybe we have different ideas about how it could work or should work.
2. I classify tests in order to better understand all the different kinds of intent and purpose we have when we write them. This helps me decide how to choose which tests to write next. What challenges do you have with all these test classifications?
3. Some people report that there are too many test classifications to understand well. They become confused. I empathize. Why don't you simply ignore those classifications until you need them?
Finally, as for the difference between business and technical tests, when I talk about TDD I tend to focus on technical tests, because that's my context for TDD: to focus on the code. I handle Customer Tests (or business tests) quite differently, and I only sometimes practise what we've called Story-TDD or Acceptance-TDD. I practise it most often when I play the role of Customer for myself, such as on my volunteer projects. I try _very hard_ to clarify this for the people I teach, but I always run the risk that the message doesn't make it through.
--
J. B. (Joe) Rainsberger :: tdd.training <>?:: jbrains.ca <> ::
blog.thecodewhisperer.com <>
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
--
J. B. (Joe) Rainsberger :: <> :: <> :: <>
Teaching evolutionary design and TDD since 2002
--
----------------------------------------------------------------------
* George Dinwiddie *
Software Development
Consultant and Coach
----------------------------------------------------------------------
|
In the past two years, where I have been working on a codebase with hundreds of thousands of tests and almost a hundred different teams touching it, I started to “care less” about semantically classifying tests. That, team members can come up with an agreement of what makes more sense to them in their context. Do we really need to have a single classification company-wise?
Nowadays, I really care about classifying tests in terms of their infrastructure costs. This matters globally and must defined at company level, because although code isn’t (well, sometimes is) shared among teams, resources are. Reliability is also another category that I care. Your want the tests you run in the pre-merge to give you 100% sound signal.
Do we allow multithreading in our unit test suite, or should these tests be somewhere else? Do we allow mock server in it? When do we need to run all the tests and when can we just run a subset of them? How can we bring (costly) integration tests to the pre-merge? What to do with flaky tests; should we delete them, should we keep them there waiting for someone to fix them, should we move them to another place? These are questions that have been on my mind when I talk about segregating tests.
Cheers, Mauricio?
On Tue, 4 Jul 2023 at 18:19, George Dinwiddie < lists@...> wrote: I agree that the naming can be confusing because often the same name
means different things to different people. I don't get too hung up on
the naming of types of tests (though I love Gpaw's "microtests" because
it gets out of the "unit test" mire). Instead, I tray to talk about the
meaning the other person has behind the name.
When I started doing TDD, I sorted my tests into three categories:
? - "unit tests" which tested in memory code without any other dependencies
? - "database tests" which tested code dependent on the database. This
led me to using the Adapter Pattern so I could isolate my unit tests
from the database and test only the adapter against a real database.
? - "deployed tests" which required the system to be deployed in order
to run. These tended to be "story tests," though I found that by
delegating from the requirements of the app server (J2EE in those days)
to Plain Old Java Objects with the same API, I could implement most of
the story tests the same way as unit tests, so "story tests" became
another category.
Eventually I had need to call other systems beyond the database, so
those became another classification of tests, but done in the same
fashion as the database tests.
? - George
On 7/4/23 9:38 AM, J. B. Rainsberger wrote:
>? > I think the test name separation by unit test/integration test/micro
> test/behaviour test doesn't work, but I'm not sure what's the "sensible"
> way to separate them yetlike fast/slow/io/non-io?
> business/technical?structure vs behaviour?
>
> I'm curious about this, because I hear it from time to time.
>
> I have some questions, and we don't have to limit those to Tony,
> although I'm also interested in his replies:
>
> 1. What kind of "doesn't work"? It works for me, so maybe we have
> different ideas about how it could work or should work.
>
> 2. I classify tests in order to better understand all the different
> kinds of intent and purpose we have when we write them. This helps me
> decide how to choose which tests to write next. What challenges do you
> have with all these test classifications?
>
> 3. Some people report that there are too many test classifications to
> understand well. They become confused. I empathize. Why don't you simply
> ignore those classifications until you need them?
>
> Finally, as for the difference between business and technical tests,
> when I talk about TDD I tend to focus on technical tests, because that's
> my context for TDD: to focus on the code. I handle Customer Tests (or
> business tests) quite differently, and I only sometimes practise what
> we've called Story-TDD or Acceptance-TDD. I practise it most often when
> I play the role of Customer for myself, such as on my volunteer
> projects. I try _very hard_ to clarify this for the people I teach, but
> I always run the risk that the message doesn't make it through.
> --
> J. B. (Joe) Rainsberger :: tdd.training <>?::
> <> ::
> <>
>
> Replies from this account routinely take a few days, which allows me to
> reply thoughtfully. I reply more quickly to messages that clearly
> require answers urgently. If you need something from me and are on a
> deadline, then let me know how soon you need a reply so that I can
> better help you to get what you need. Thank you for your consideration.
>
> --
> J. B. (Joe) Rainsberger :: <>
> :: <>
> :: <>
> Teaching evolutionary design and TDD since 2002
>
--
? ----------------------------------------------------------------------
? ?* George Dinwiddie *? ? ? ? ? ? ? ? ? ? ?
? ?Software Development? ? ? ? ? ? ? ? ? ?
? ?Consultant and Coach? ? ? ? ?
? ----------------------------------------------------------------------
-- -- Maurício AnicheAuthor of
|
Totally worth you?Maurício there.
Having been responsible (inherited) for a 40 min test suite on an app worked on by hundreds of devs: determinism and cost were high on my priority list.
Fox ---
In the past two years, where I have been working on a codebase with hundreds of thousands of tests and almost a hundred different teams touching it, I started to “care less” about semantically classifying tests. That, team members can come up with an agreement of what makes more sense to them in their context. Do we really need to have a single classification company-wise?
Nowadays, I really care about classifying tests in terms of their infrastructure costs. This matters globally and must defined at company level, because although code isn’t (well, sometimes is) shared among teams, resources are. Reliability is also another category that I care. Your want the tests you run in the pre-merge to give you 100% sound signal.
Do we allow multithreading in our unit test suite, or should these tests be somewhere else? Do we allow mock server in it? When do we need to run all the tests and when can we just run a subset of them? How can we bring (costly) integration tests to the pre-merge? What to do with flaky tests; should we delete them, should we keep them there waiting for someone to fix them, should we move them to another place? These are questions that have been on my mind when I talk about segregating tests.
Cheers, Mauricio?
On Tue, 4 Jul 2023 at 18:19, George Dinwiddie < lists@...> wrote: I agree that the naming can be confusing because often the same name
means different things to different people. I don't get too hung up on
the naming of types of tests (though I love Gpaw's "microtests" because
it gets out of the "unit test" mire). Instead, I tray to talk about the
meaning the other person has behind the name.
When I started doing TDD, I sorted my tests into three categories:
? - "unit tests" which tested in memory code without any other dependencies
? - "database tests" which tested code dependent on the database. This
led me to using the Adapter Pattern so I could isolate my unit tests
from the database and test only the adapter against a real database.
? - "deployed tests" which required the system to be deployed in order
to run. These tended to be "story tests," though I found that by
delegating from the requirements of the app server (J2EE in those days)
to Plain Old Java Objects with the same API, I could implement most of
the story tests the same way as unit tests, so "story tests" became
another category.
Eventually I had need to call other systems beyond the database, so
those became another classification of tests, but done in the same
fashion as the database tests.
? - George
On 7/4/23 9:38 AM, J. B. Rainsberger wrote:
>? > I think the test name separation by unit test/integration test/micro
> test/behaviour test doesn't work, but I'm not sure what's the "sensible"
> way to separate them yetlike fast/slow/io/non-io?
> business/technical?structure vs behaviour?
>
> I'm curious about this, because I hear it from time to time.
>
> I have some questions, and we don't have to limit those to Tony,
> although I'm also interested in his replies:
>
> 1. What kind of "doesn't work"? It works for me, so maybe we have
> different ideas about how it could work or should work.
>
> 2. I classify tests in order to better understand all the different
> kinds of intent and purpose we have when we write them. This helps me
> decide how to choose which tests to write next. What challenges do you
> have with all these test classifications?
>
> 3. Some people report that there are too many test classifications to
> understand well. They become confused. I empathize. Why don't you simply
> ignore those classifications until you need them?
>
> Finally, as for the difference between business and technical tests,
> when I talk about TDD I tend to focus on technical tests, because that's
> my context for TDD: to focus on the code. I handle Customer Tests (or
> business tests) quite differently, and I only sometimes practise what
> we've called Story-TDD or Acceptance-TDD. I practise it most often when
> I play the role of Customer for myself, such as on my volunteer
> projects. I try _very hard_ to clarify this for the people I teach, but
> I always run the risk that the message doesn't make it through.
> --
> J. B. (Joe) Rainsberger :: tdd.training <>?::
> <> ::
> <>
>
> Replies from this account routinely take a few days, which allows me to
> reply thoughtfully. I reply more quickly to messages that clearly
> require answers urgently. If you need something from me and are on a
> deadline, then let me know how soon you need a reply so that I can
> better help you to get what you need. Thank you for your consideration.
>
> --
> J. B. (Joe) Rainsberger :: <>
> :: <>
> :: <>
> Teaching evolutionary design and TDD since 2002
>
--
? ----------------------------------------------------------------------
? ?* George Dinwiddie *? ? ? ? ? ? ? ? ? ? ?
? ?Software Development? ? ? ? ? ? ? ? ? ?
? ?Consultant and Coach? ? ? ? ?
? ----------------------------------------------------------------------
-- Maurício AnicheAuthor of
|
I have similar ideas. Classifying tests in terms of where in the pipeline they are most helpful in terms of ensuring quality makes most sense to me.
That classification?would be something?like "build", "pre-push", "pre-merge", "pre-deploy", "post-deploy" and "monitoring-production" (periodic runs reporting to a monitoring tool).?
In the past two years, where I have been working on a codebase with hundreds of thousands of tests and almost a hundred different teams touching it, I started to “care less” about semantically classifying tests. That, team members can come up with an agreement of what makes more sense to them in their context. Do we really need to have a single classification company-wise?
Nowadays, I really care about classifying tests in terms of their infrastructure costs. This matters globally and must defined at company level, because although code isn’t (well, sometimes is) shared among teams, resources are. Reliability is also another category that I care. Your want the tests you run in the pre-merge to give you 100% sound signal.
Do we allow multithreading in our unit test suite, or should these tests be somewhere else? Do we allow mock server in it? When do we need to run all the tests and when can we just run a subset of them? How can we bring (costly) integration tests to the pre-merge? What to do with flaky tests; should we delete them, should we keep them there waiting for someone to fix them, should we move them to another place? These are questions that have been on my mind when I talk about segregating tests.
Cheers, Mauricio?
On Tue, 4 Jul 2023 at 18:19, George Dinwiddie < lists@...> wrote: I agree that the naming can be confusing because often the same name
means different things to different people. I don't get too hung up on
the naming of types of tests (though I love Gpaw's "microtests" because
it gets out of the "unit test" mire). Instead, I tray to talk about the
meaning the other person has behind the name.
When I started doing TDD, I sorted my tests into three categories:
? - "unit tests" which tested in memory code without any other dependencies
? - "database tests" which tested code dependent on the database. This
led me to using the Adapter Pattern so I could isolate my unit tests
from the database and test only the adapter against a real database.
? - "deployed tests" which required the system to be deployed in order
to run. These tended to be "story tests," though I found that by
delegating from the requirements of the app server (J2EE in those days)
to Plain Old Java Objects with the same API, I could implement most of
the story tests the same way as unit tests, so "story tests" became
another category.
Eventually I had need to call other systems beyond the database, so
those became another classification of tests, but done in the same
fashion as the database tests.
? - George
On 7/4/23 9:38 AM, J. B. Rainsberger wrote:
>? > I think the test name separation by unit test/integration test/micro
> test/behaviour test doesn't work, but I'm not sure what's the "sensible"
> way to separate them yetlike fast/slow/io/non-io?
> business/technical?structure vs behaviour?
>
> I'm curious about this, because I hear it from time to time.
>
> I have some questions, and we don't have to limit those to Tony,
> although I'm also interested in his replies:
>
> 1. What kind of "doesn't work"? It works for me, so maybe we have
> different ideas about how it could work or should work.
>
> 2. I classify tests in order to better understand all the different
> kinds of intent and purpose we have when we write them. This helps me
> decide how to choose which tests to write next. What challenges do you
> have with all these test classifications?
>
> 3. Some people report that there are too many test classifications to
> understand well. They become confused. I empathize. Why don't you simply
> ignore those classifications until you need them?
>
> Finally, as for the difference between business and technical tests,
> when I talk about TDD I tend to focus on technical tests, because that's
> my context for TDD: to focus on the code. I handle Customer Tests (or
> business tests) quite differently, and I only sometimes practise what
> we've called Story-TDD or Acceptance-TDD. I practise it most often when
> I play the role of Customer for myself, such as on my volunteer
> projects. I try _very hard_ to clarify this for the people I teach, but
> I always run the risk that the message doesn't make it through.
> --
> J. B. (Joe) Rainsberger :: tdd.training <>?::
> <> ::
> <>
>
> Replies from this account routinely take a few days, which allows me to
> reply thoughtfully. I reply more quickly to messages that clearly
> require answers urgently. If you need something from me and are on a
> deadline, then let me know how soon you need a reply so that I can
> better help you to get what you need. Thank you for your consideration.
>
> --
> J. B. (Joe) Rainsberger :: <>
> :: <>
> :: <>
> Teaching evolutionary design and TDD since 2002
>
--
? ----------------------------------------------------------------------
? ?* George Dinwiddie *? ? ? ? ? ? ? ? ? ? ?
? ?Software Development? ? ? ? ? ? ? ? ? ?
? ?Consultant and Coach? ? ? ? ?
? ----------------------------------------------------------------------
-- Maurício AnicheAuthor of
|
I'm a bear of little brain.?
There are tests that are fast & good enough to support refactoring (only, maybe): you can happily run them after renaming a variable.
There are tests that show that the system's functions work, and which aren't fast enough to run them after a variable rename.
Both are important. Ultimately there are "the system works" and there are "the refactoring step didn't break anything" and the rest is just intent and technology.
In the past two years, where I have been working on a codebase with hundreds of thousands of tests and almost a hundred different teams touching it, I started to “care less” about semantically classifying tests. That, team members can come up with an agreement of what makes more sense to them in their context. Do we really need to have a single classification company-wise?
Nowadays, I really care about classifying tests in terms of their infrastructure costs. This matters globally and must defined at company level, because although code isn’t (well, sometimes is) shared among teams, resources are. Reliability is also another category that I care. Your want the tests you run in the pre-merge to give you 100% sound signal.
Do we allow multithreading in our unit test suite, or should these tests be somewhere else? Do we allow mock server in it? When do we need to run all the tests and when can we just run a subset of them? How can we bring (costly) integration tests to the pre-merge? What to do with flaky tests; should we delete them, should we keep them there waiting for someone to fix them, should we move them to another place? These are questions that have been on my mind when I talk about segregating tests.
Cheers, Mauricio?
On Tue, 4 Jul 2023 at 18:19, George Dinwiddie < lists@...> wrote: I agree that the naming can be confusing because often the same name
means different things to different people. I don't get too hung up on
the naming of types of tests (though I love Gpaw's "microtests" because
it gets out of the "unit test" mire). Instead, I tray to talk about the
meaning the other person has behind the name.
When I started doing TDD, I sorted my tests into three categories:
? - "unit tests" which tested in memory code without any other dependencies
? - "database tests" which tested code dependent on the database. This
led me to using the Adapter Pattern so I could isolate my unit tests
from the database and test only the adapter against a real database.
? - "deployed tests" which required the system to be deployed in order
to run. These tended to be "story tests," though I found that by
delegating from the requirements of the app server (J2EE in those days)
to Plain Old Java Objects with the same API, I could implement most of
the story tests the same way as unit tests, so "story tests" became
another category.
Eventually I had need to call other systems beyond the database, so
those became another classification of tests, but done in the same
fashion as the database tests.
? - George
On 7/4/23 9:38 AM, J. B. Rainsberger wrote:
>? > I think the test name separation by unit test/integration test/micro
> test/behaviour test doesn't work, but I'm not sure what's the "sensible"
> way to separate them yetlike fast/slow/io/non-io?
> business/technical?structure vs behaviour?
>
> I'm curious about this, because I hear it from time to time.
>
> I have some questions, and we don't have to limit those to Tony,
> although I'm also interested in his replies:
>
> 1. What kind of "doesn't work"? It works for me, so maybe we have
> different ideas about how it could work or should work.
>
> 2. I classify tests in order to better understand all the different
> kinds of intent and purpose we have when we write them. This helps me
> decide how to choose which tests to write next. What challenges do you
> have with all these test classifications?
>
> 3. Some people report that there are too many test classifications to
> understand well. They become confused. I empathize. Why don't you simply
> ignore those classifications until you need them?
>
> Finally, as for the difference between business and technical tests,
> when I talk about TDD I tend to focus on technical tests, because that's
> my context for TDD: to focus on the code. I handle Customer Tests (or
> business tests) quite differently, and I only sometimes practise what
> we've called Story-TDD or Acceptance-TDD. I practise it most often when
> I play the role of Customer for myself, such as on my volunteer
> projects. I try _very hard_ to clarify this for the people I teach, but
> I always run the risk that the message doesn't make it through.
> --
> J. B. (Joe) Rainsberger :: tdd.training <>?::
> <> ::
> <>
>
> Replies from this account routinely take a few days, which allows me to
> reply thoughtfully. I reply more quickly to messages that clearly
> require answers urgently. If you need something from me and are on a
> deadline, then let me know how soon you need a reply so that I can
> better help you to get what you need. Thank you for your consideration.
>
> --
> J. B. (Joe) Rainsberger :: <>
> :: <>
> :: <>
> Teaching evolutionary design and TDD since 2002
>
--
? ----------------------------------------------------------------------
? ?* George Dinwiddie *? ? ? ? ? ? ? ? ? ? ?
? ?Software Development? ? ? ? ? ? ? ? ? ?
? ?Consultant and Coach? ? ? ? ?
? ----------------------------------------------------------------------
-- Maurício AnicheAuthor of
-- Peace, Tim
-------------------------------------
|
I see those as: 1) tests that ensure that the code does what the developers intend it to do, and 2) tests that ensure that the code does what the users expect it to do.
toggle quoted message
Show quoted text
On Jul 6, 2023, at 12:22 PM, Tim Ottinger <tottinge@...> wrote:
I'm a bear of little brain.?
There are tests that are fast & good enough to support refactoring (only, maybe): you can happily run them after renaming a variable.
There are tests that show that the system's functions work, and which aren't fast enough to run them after a variable rename.
Both are important. Ultimately there are "the system works" and there are "the refactoring step didn't break anything" and the rest is just intent and technology. In the past two years, where I have been working on a codebase with hundreds of thousands of tests and almost a hundred different teams touching it, I started to “care less” about semantically classifying tests. That, team members can come up with an agreement of what makes more sense to them in their context. Do we really need to have a single classification company-wise?
Nowadays, I really care about classifying tests in terms of their infrastructure costs. This matters globally and must defined at company level, because although code isn’t (well, sometimes is) shared among teams, resources are. Reliability is also another category that I care. Your want the tests you run in the pre-merge to give you 100% sound signal.
Do we allow multithreading in our unit test suite, or should these tests be somewhere else? Do we allow mock server in it? When do we need to run all the tests and when can we just run a subset of them? How can we bring (costly) integration tests to the pre-merge? What to do with flaky tests; should we delete them, should we keep them there waiting for someone to fix them, should we move them to another place? These are questions that have been on my mind when I talk about segregating tests.
Cheers, Mauricio?
On Tue, 4 Jul 2023 at 18:19, George Dinwiddie < lists@...> wrote: I agree that the naming can be confusing because often the same name?
means different things to different people. I don't get too hung up on?
the naming of types of tests (though I love Gpaw's "microtests" because?
it gets out of the "unit test" mire). Instead, I tray to talk about the?
meaning the other person has behind the name.
When I started doing TDD, I sorted my tests into three categories:
??- "unit tests" which tested in memory code without any other dependencies
??- "database tests" which tested code dependent on the database. This?
led me to using the Adapter Pattern so I could isolate my unit tests?
from the database and test only the adapter against a real database.
??- "deployed tests" which required the system to be deployed in order?
to run. These tended to be "story tests," though I found that by?
delegating from the requirements of the app server (J2EE in those days)?
to Plain Old Java Objects with the same API, I could implement most of?
the story tests the same way as unit tests, so "story tests" became?
another category.
Eventually I had need to call other systems beyond the database, so?
those became another classification of tests, but done in the same?
fashion as the database tests.
??- George
On 7/4/23 9:38 AM, J. B. Rainsberger wrote:
>? > I think the test name separation by unit test/integration test/micro?
> test/behaviour test doesn't work, but I'm not sure what's the "sensible"?
> way to separate them yetlike fast/slow/io/non-io??
> business/technical?structure vs behaviour?
>?
> I'm curious about this, because I hear it from time to time.
>?
> I have some questions, and we don't have to limit those to Tony,?
> although I'm also interested in his replies:
>?
> 1. What kind of "doesn't work"? It works for me, so maybe we have?
> different ideas about how it could work or should work.
>?
> 2. I classify tests in order to better understand all the different?
> kinds of intent and purpose we have when we write them. This helps me?
> decide how to choose which tests to write next. What challenges do you?
> have with all these test classifications?
>?
> 3. Some people report that there are too many test classifications to?
> understand well. They become confused. I empathize. Why don't you simply?
> ignore those classifications until you need them?
>?
> Finally, as for the difference between business and technical tests,?
> when I talk about TDD I tend to focus on technical tests, because that's?
> my context for TDD: to focus on the code. I handle Customer Tests (or?
> business tests) quite differently, and I only sometimes practise what?
> we've called Story-TDD or Acceptance-TDD. I practise it most often when?
> I play the role of Customer for myself, such as on my volunteer?
> projects. I try _very hard_ to clarify this for the people I teach, but?
> I always run the risk that the message doesn't make it through.
> --?
> J. B. (Joe) Rainsberger :: tdd.training <>?::?
>??<> ::
>??<>
>?
> Replies from this account routinely take a few days, which allows me to?
> reply thoughtfully. I reply more quickly to messages that clearly?
> require answers urgently. If you need something from me and are on a?
> deadline, then let me know how soon you need a reply so that I can?
> better help you to get what you need. Thank you for your consideration.
>?
> --?
> J. B. (Joe) Rainsberger ::??<>?
> ::??<>?
> ::??<>
> Teaching evolutionary design and TDD since 2002
>?
--?
??----------------------------------------------------------------------
? ?* George Dinwiddie *? ? ? ? ? ? ? ? ? ? ??
? ?Software Development? ? ? ? ? ? ? ? ? ??
? ?Consultant and Coach? ? ? ? ?
??----------------------------------------------------------------------
-- Maurício AnicheAuthor of?
--?Peace, Tim
-------------------------------------
|
Rather than categorical classification, perhaps it is useful to think of "axes" of test strategy (plural of "axis", not "axe"):
The main ones in my mind are... 1. Collaboration: How many production objects/classes/modules are exercised in the test? 2. I/O: How much access to?disk/network/clock/etc. happens during the test? ... as well as: 3. Doubles: How many interactions with test versions of other objects/classes/modules does the test need to set up to do its job? How complex is the usage of test doubles?
I think of (3) as mostly something to be minimized(*), whereas there are legit tradeoffs along the first two axes.
(*) "minimized" does not mean "completely avoided".
Some combinations of these qualities, or "points" in my imaginary 3D testing strategy problem space:
Low collaboration, no I/O, no doubles:? What almost everyone would call a "unit" or "isolated test".? High design pressure on unit boundaries. The code under test is essentially a pure function -- the "core" of functional core/imperative shell design.
Low collaboration, no I/O, with doubles:? When testing impure production code, there is likely some setup (axis 3). High design pressure, high coupling. Examples: integrations with the outside?world. Valuable if done well, very easy to do poorly.
Low collaboration, with I/O Tests of adapters to owned/easy to setup up resources (ie database wrappers) Low design pressure. Valuable in small quantity, can be costly to maintain.
High collaboration, low/no I/O Tests of subsystems, often with fakes of modules that use I/O in production. Low design pressure, low coupling, high support for structural refactoring. This is one place where test doubles?(fakes! not mocks or spies!) are very useful in my practice. Extremely valuable and underused in my opinion. That said, there is a danger of neglecting lower-level design/testing.
High collaboration, high I/O "End to end" or "integration tests". No design pressure. Danger of neglecting lower-level design/testing. False sense of security. Fool's gold. "Easy" to write, sort of, brittle, provide no design guidance. WAY overused in my experience.
On Tue, Jul 4, 2023 at 6:39?AM J. B. Rainsberger < me@...> wrote: > I think the test name separation by unit test/integration test/micro
test/behaviour test doesn't work, but I'm not sure what's the "sensible"
way to separate them yet like fast/slow/io/non-io? business/technical?structure vs behaviour?
I'm curious about this, because I hear it from time to time.
I have some questions, and we don't have to limit those to Tony, although I'm also interested in his replies:
1. What kind of "doesn't work"? It works for me, so maybe we have different ideas about how it could work or should work.
2. I classify tests in order to better understand all the different kinds of intent and purpose we have when we write them. This helps me decide how to choose which tests to write next. What challenges do you have with all these test classifications?
3. Some people report that there are too many test classifications to understand well. They become confused. I empathize. Why don't you simply ignore those classifications until you need them?
Finally, as for the difference between business and technical tests, when I talk about TDD I tend to focus on technical tests, because that's my context for TDD: to focus on the code. I handle Customer Tests (or business tests) quite differently, and I only sometimes practise what we've called Story-TDD or Acceptance-TDD. I practise it most often when I play the role of Customer for myself, such as on my volunteer projects. I try _very hard_ to clarify this for the people I teach, but I always run the risk that the message doesn't make it through. -- J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
--
J. B. (Joe) Rainsberger :: :: ::
Teaching evolutionary design and TDD since 2002
|
Possibly that is why in XP we called them programmer tests and customer tests. :) On Jul 6, 2023, at 12:28 PM, Russell Gold <russ@...> wrote:
I see those as:1) tests that ensure that the code does what the developers intend it to do, and 2) tests that ensure that the code does what the users expect it to do.
Ron Jeffries
If it is more than you need, it is waste. -- Andy Seidl
|
On Tue, Jul 4, 2023 at 1:19?PM George Dinwiddie < lists@...> wrote: ? I agree that the naming can be confusing because often the same name
means different things to different people. I don't get too hung up on
the naming of types of tests (though I love Gpaw's "microtests" because
it gets out of the "unit test" mire). Instead, I tray to talk about the
meaning the other person has behind the name.
I'd like to second this loudly: I noticed a huge improvement in my relationships with people when I stopped insisting on my understanding of the terms and instead focused on exploring the differences in our mutual understanding of the terms. J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
-- J. B. (Joe) Rainsberger :: :: :: Teaching evolutionary design and TDD since 2002
|
? In the past two years, where I have been working on a codebase with hundreds of thousands of tests and almost a hundred different teams touching it, I started to “care less” about semantically classifying tests. That, team members can come up with an agreement of what makes more sense to them in their context. Do we really need to have a single classification company-wise?
It might be wise to have a common vocabulary among such a large project community, but I definitely don't recommend One Corporation-Wide Strategy. :) ?
Nowadays, I really care about classifying tests in terms of their infrastructure costs. This matters globally and must defined at company level, because although code isn’t (well, sometimes is) shared among teams, resources are. Reliability is also another category that I care. Your want the tests you run in the pre-merge to give you 100% sound signal.
I love it! "Let's classify based on what actually matters to us" instead of "Let's classify based on some abstract notion of correctness that some person wrote in a book one time". ?
Do we allow multithreading in our unit test suite, or should these tests be somewhere else? Do we allow mock server in it? When do we need to run all the tests and when can we just run a subset of them? How can we bring (costly) integration tests to the pre-merge? What to do with flaky tests; should we delete them, should we keep them there waiting for someone to fix them, should we move them to another place? These are questions that have been on my mind when I talk about segregating tests.
This reminds me of the first project in which I participated with other people practising TDD who actually wanted to do it. We simply posted the top 10 list of slowest tests every day and used that to guide our choices about breaking integrated things apart to make them more individually inspectable. That was enough. J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
-- J. B. (Joe) Rainsberger :: :: :: Teaching evolutionary design and TDD since 2002
|
? Possibly that is why in XP we called them programmer tests and customer tests. :)
That's one usually-quite-important axis. :) It's one of the ones I continue to consider regularly.
J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
-- J. B. (Joe) Rainsberger :: :: :: Teaching evolutionary design and TDD since 2002
|
I like examples.
WRT naming, I worked at an insurance company. There were problems with churn in API code, particularly around certain domain concepts.
One of those was 'customer'.
To the policy lifecycle team, customer meant 'a person that has bought one of our policies'.
To the sales team, customer meant 'someone who is considering buying one of our policies'.
To the support team, customer meant 'a person who needs help, who may or may not hold a current or expired policy'.
Needless to say, the requirements over what data these teams needed to store about a 'customer' were quite different, although there was (of course) common elements.
Once everyone understood that these were different things, we were able to agree on different language in order to differentiate different groups' concerns.
Fox ---
On Wed, 26 Jul 2023, 15:19 J. B. Rainsberger, < me@...> wrote: ? Possibly that is why in XP we called them programmer tests and customer tests. :)
That's one usually-quite-important axis. :) It's one of the ones I continue to consider regularly.
J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
--
J. B. (Joe) Rainsberger :: :: ::
Teaching evolutionary design and TDD since 2002
|
In the context of software development, a customer test?refers to a specification?of a single thing that the people paying for the software want the software to do in the form of an executable test.??
While this sense of customer is not directly related to what customer might mean in the domain model for the software, it can have similar ambiguities and nuances because the people paying for the software usually delegate that level of specification to some combination of line managers, users, business analysts, product marketing, product management, sales, product owners or even the same team that is writing the programmer tests.
I like examples.
WRT naming, I worked at an insurance company. There were problems with churn in API code, particularly around certain domain concepts.
One of those was 'customer'.
To the policy lifecycle team, customer meant 'a person that has bought one of our policies'.
To the sales team, customer meant 'someone who is considering buying one of our policies'.
To the support team, customer meant 'a person who needs help, who may or may not hold a current or expired policy'.
Needless to say, the requirements over what data these teams needed to store about a 'customer' were quite different, although there was (of course) common elements.
Once everyone understood that these were different things, we were able to agree on different language in order to differentiate different groups' concerns.
Fox ---
On Wed, 26 Jul 2023, 15:19 J. B. Rainsberger, < me@...> wrote: ? Possibly that is why in XP we called them programmer tests and customer tests. :)
That's one usually-quite-important axis. :) It's one of the ones I continue to consider regularly.
J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
--
J. B. (Joe) Rainsberger :: :: ::
Teaching evolutionary design and TDD since 2002
|
I've often worked with (and in) organizations where the perfect word (from my point of view) for a concept was already in use for something else. I've found that it useless to try to change existing usage, and confusing to ignore existing usage. Choosing another word to use in that context is well worth the trouble, even if that word seems a little clumsy.
- George
toggle quoted message
Show quoted text
On 7/28/23 4:44 PM, Steve Gordon wrote: In the context of software development, a customer test?refers to a specification?of a single thing that the people paying for the software want the software to do in the form of an executable test.
While this sense of customer is not directly related to what customer might mean in the domain model for the software, it can have similar ambiguities and nuances because the people paying for the software usually delegate that level of specification to some combination of line managers, users, business analysts, product marketing, product management, sales, product owners or even the same team that is writing the programmer tests.
On Fri, Jul 28, 2023 at 1:06?PM Sleepyfox <sleepyfox@... <mailto:sleepyfox@...>> wrote:
I like examples.
WRT naming, I worked at an insurance company. There were problems
with churn in API code, particularly around certain domain concepts.
One of those was 'customer'.
To the policy lifecycle team, customer meant 'a person that has
bought one of our policies'.
To the sales team, customer meant 'someone who is considering buying
one of our policies'.
To the support team, customer meant 'a person who needs help, who
may or may not hold a current or expired policy'.
Needless to say, the requirements over what data these teams needed
to store about a 'customer' were quite different, although there was
(of course) common elements.
Once everyone understood that these were different things, we were
able to agree on different language in order to differentiate
different groups' concerns.
Fox
---
On Wed, 26 Jul 2023, 15:19 J. B. Rainsberger, <me@...
<mailto:me@...>> wrote:
On Thu, Jul 6, 2023 at 8:46?PM Ron Jeffries <ronjeffries@...
<mailto:ronjeffries@...>> wrote:
Possibly that is why in XP we called them programmer tests
and customer tests. :)
That's one usually-quite-important axis. :) It's one of the ones
I continue to consider regularly.
--
J. B. (Joe) Rainsberger :: tdd.training
<>?:: jbrains.ca <> ::
blog.thecodewhisperer.com <>
Replies from this account routinely take a few days, which
allows me to reply thoughtfully. I reply more quickly to
messages that clearly require answers urgently. If you need
something from me and are on a deadline, then let me know how
soon you need a reply so that I can better help you to get what
you need. Thank you for your consideration.
--
J. B. (Joe) Rainsberger ::
<> ::
<> ::
<>
Teaching evolutionary design and TDD since 2002
--
----------------------------------------------------------------------
* George Dinwiddie *
Software Development
Consultant and Coach
----------------------------------------------------------------------
|
That is some WIZE advice…. Wish I could have heard it 20 yrs ago…. I’d not have banged mine head so much and raged against the improper use of a term!
toggle quoted message
Show quoted text
On Jul 29, 2023, at 9:58 AM, George Dinwiddie <lists@...> wrote:
I've found that it useless to try to change existing usage, and confusing to ignore existing usage.
|
David,
I got a few lumps on the noggin learning that.
- George
toggle quoted message
Show quoted text
On 7/29/23 1:44 PM, David Koontz wrote: That is some WIZE advice…. Wish I could have heard it 20 yrs ago…. I’d not have banged mine head so much and raged against the improper use of a term!
On Jul 29, 2023, at 9:58 AM, George Dinwiddie <lists@...> wrote:
I've found that it useless to try to change existing usage, and confusing to ignore existing usage.
--
----------------------------------------------------------------------
* George Dinwiddie *
Software Development
Consultant and Coach
----------------------------------------------------------------------
|
Wisdom, for me, includes no longer viewing such usage as "improper", but merely something more like "unexpected" or "different". I don't even mean that we stop calling it improper, but that we stop experiencing it as improper. :) Many do the first; fewer do the second.
On Sat, Jul 29, 2023 at 10:45?AM David Koontz < david@...> wrote: That is some WIZE advice…. Wish I could have heard it 20 yrs ago…. I’d not have banged mine head so much and raged against the improper use of a term! On Jul 29, 2023, at 9:58 AM, George Dinwiddie < lists@...> wrote:
I've found that it useless to try to change existing usage, and confusing to ignore existing usage.
J. B. (Joe) Rainsberger :: ?:: ::
Replies from this account routinely take a few days, which allows me to reply thoughtfully. I reply more quickly to messages that clearly require answers urgently. If you need something from me and are on a deadline, then let me know how soon you need a reply so that I can better help you to get what you need. Thank you for your consideration.
--
J. B. (Joe) Rainsberger :: :: ::
Teaching evolutionary design and TDD since 2002
|
J. B. Rainsberger
Sleepyfox
Gu?laugur Egilsson
David Koontz