Are you familiar with Command Query Separation (CQS)? I've always found that the most coherent and useful way to explain why you would choose a mock or a stub. The most interesting thing about a method is either what it returns or, in the case where you don't care what it returns, whether it was called with the correct semantics, not both.?

So, if the SUT cares about the value that a method returns then you stub that method, only. If the SUT doesn't care about the value a method returns then you use a mock and verify the call has the expected semantics, only.?

Does that help??

On Tue, Jan 14, 2014 at 6:37 AM, Mateusz ?oskot <mateusz@...> wrote:

?

Hi,

I'd like to ask for ideas of best practices in case of developing and testing
a wrapper for a Python legacy module.
Especially, about applying the single assert / single mock rule
which I have been learning about from numerous books/videos/blogs.

I have Python xdep module which exposes API in form of bunch of free functions.
I'm creating new Python module foo which uses xdep (the idea is to provide
simplified, Pythonic interface, but that is not directly relevant to
my question here).

I'm trying to keep the code as simple & generic as possible, so my question is
not about Python but unit testing and mocking.

# xdep.py
def open():
...
def get_item(item_id):
...
def close():
...

# foo.py
import xdep

def exists(item_id):
xdep.open()
item = xdep.get_item(item_id)
xdep.close()
if item:
return == item_id
else:
return false

Here is my module to unit test behaviuour of the foo:
- I create and plug a stub for the legacy module xdep
- I mock functions of xdep

# test_foo.py
import unittest
import unittest.mock as mock

class ItemTest(unittest.TestCase):
def setUp(self):

# create stub for external dependency of foo
self.xdep_stub = mock.Mock()
patcher = mock.patch.dict('sys.modules', {'xdep': self.xdep_stub})
self.addCleanup(patcher.stop)
patcher.start()

# import foo does import xdep, as patched above
import foo
self.foo = foo

def test_exists_ExistingItemId_ReturnTrue(self):
foo = self.foo
# Arrange
### Data
item_id = 1
### Mock
foo.xdep.get_item.return_value = 1
# Act
item_exists = foo.item.exists(item_id)
# Assert
self.assertTrue(item_exists)

I have a single assertion and I use single mock which mocks xdep.get_item
function, so it seems a valid yet clear unit test.

Now, having the single assert/single mock rule in mind, I have some questions,
about the test_exists_ExistingItemId_ReturnTrue in particular:

1) How asserting on foo.item.exists return value is different from
asserting how xdep.get_item mock was called?

I could have replaced
self.assertTrue(item_exists)
with
foo.xdep.get_item.assert_called_once_with(1)

2) What about having both assertions, would that break the single
assertion rule?

self.assertTrue(item_exists)
foo.xdep.get_item.assert_called_once_with(1)

I'm not concern about breaking the rule as technical convention, but as:
am I testing multiple things here?

3) AFAIU, it's natural that unit tests are coupled with implementation of
method/behaviour they are testing, foo.exists function in my case.

So, given that foo.exists function calls three functions of the legacy
xdep module
xdep.open
xdep.get_item
xdep.close
should I mock all the three function and verify how they were called?
Would that still belong to scope of test_exists_ExistingItemId_ReturnTrue test
or I better create a new dedicated test case(s), something like
test_exists_IfCalled_xdepOpenCalled
test_exists_IfCalled_xdepCloseCalled
test_exists_IfCalled_xdepGetItemCalled
and do assert_called_once_with() appropriately.

Above, I count each mock for each xdep function as a separate mock,
so "single mock per test" rule seems to suggests I should verify they are called
in separate tests, doesn't it?

Also, some of other xdep functions need to be called with multiple parameters,
and as setting up and verifying their mocks may be quite elaborate,my
gut feeling
tells me that single unit test per "legacy API call expectation" as listed above
is a good approach for readability and maintainability of tests.
Am I right?

4) What other, if any, approach to single assert/single mock rules
would you take for testing similar case as above?

Best regards,
--
Mateusz ?oskot,

?

Hi,

I'd like to ask for ideas of best practices in case of developing and testing
a wrapper for a Python legacy module.
Especially, about applying the single assert / single mock rule
which I have been learning about from numerous books/videos/blogs.

I have Python xdep module which exposes API in form of bunch of free functions.
I'm creating new Python module foo which uses xdep (the idea is to provide
simplified, Pythonic interface, but that is not directly relevant to
my question here).

I'm trying to keep the code as simple & generic as possible, so my question is
not about Python but unit testing and mocking.

# xdep.py
def open():
...
def get_item(item_id):
...
def close():
...

# foo.py
import xdep

def exists(item_id):
xdep.open()
item = xdep.get_item(item_id)
xdep.close()
if item:
return == item_id
else:
return false

Here is my module to unit test behaviuour of the foo:
- I create and plug a stub for the legacy module xdep
- I mock functions of xdep

# test_foo.py
import unittest
import unittest.mock as mock

class ItemTest(unittest.TestCase):
def setUp(self):

# create stub for external dependency of foo
self.xdep_stub = mock.Mock()
patcher = mock.patch.dict('sys.modules', {'xdep': self.xdep_stub})
self.addCleanup(patcher.stop)
patcher.start()

# import foo does import xdep, as patched above
import foo
self.foo = foo

def test_exists_ExistingItemId_ReturnTrue(self):
foo = self.foo
# Arrange
### Data
item_id = 1
### Mock
foo.xdep.get_item.return_value = 1
# Act
item_exists = foo.item.exists(item_id)
# Assert
self.assertTrue(item_exists)

I have a single assertion and I use single mock which mocks xdep.get_item
function, so it seems a valid yet clear unit test.

Now, having the single assert/single mock rule in mind, I have some questions,
about the test_exists_ExistingItemId_ReturnTrue in particular:

1) How asserting on foo.item.exists return value is different from
asserting how xdep.get_item mock was called?

I could have replaced
self.assertTrue(item_exists)
with
foo.xdep.get_item.assert_called_once_with(1)

2) What about having both assertions, would that break the single
assertion rule?

self.assertTrue(item_exists)
foo.xdep.get_item.assert_called_once_with(1)

I'm not concern about breaking the rule as technical convention, but as:
am I testing multiple things here?

3) AFAIU, it's natural that unit tests are coupled with implementation of
method/behaviour they are testing, foo.exists function in my case.

So, given that foo.exists function calls three functions of the legacy
xdep module
xdep.open
xdep.get_item
xdep.close
should I mock all the three function and verify how they were called?
Would that still belong to scope of test_exists_ExistingItemId_ReturnTrue test
or I better create a new dedicated test case(s), something like
test_exists_IfCalled_xdepOpenCalled
test_exists_IfCalled_xdepCloseCalled
test_exists_IfCalled_xdepGetItemCalled
and do assert_called_once_with() appropriately.

Above, I count each mock for each xdep function as a separate mock,
so "single mock per test" rule seems to suggests I should verify they are called
in separate tests, doesn't it?

Also, some of other xdep functions need to be called with multiple parameters,
and as setting up and verifying their mocks may be quite elaborate,my
gut feeling
tells me that single unit test per "legacy API call expectation" as listed above
is a good approach for readability and maintainability of tests.
Am I right?

4) What other, if any, approach to single assert/single mock rules
would you take for testing similar case as above?

Best regards,
--
Mateusz ?oskot,

?

On 14 January 2014 16:31, Avi Kessner <akessner@...> wrote:

"I could have replaced
self.assertTrue(item_exists)
with
foo.xdep.get_item.assert_called_once_with(1)"

These two things are not?equivalent?tests.

Yes, I've come up to this conclusion after some time.

?

your first assert, is correct and is testing the interface. ?You expect the item to exist if get_item returns the value of 1.
Your second assertion is testing the implementation.

Indeed, and that is a matter of my concern.

I'm testing the implementation and somewhat implicitly I'm assuming this particular test

verifies the foo.exists() - as the foo.exists() is implemented in such way that if get_item returns?

id equal to the id that is being tested.

?

Yes, I agree it is an unfortunate test, or it is a different test, hence my questions.

What if you later change the get_item function to return an object instead of the item id? ?Do you really want to re-write this test?

Yes, I will have to re-write the test and as far as I understand it is nothing uncommon.

I admit, this also confuses me a bit.

Isn't a unit test somewhat tightly coupled to implementation?

Isn't the fact that I create a mock for a dependency used deep in

implementation of method/behaviour under test, isn't it exposing

implementation details, hence coupling the test and increasing risk

of reimplementing test if the method/behaviour under test changes?

AFAIU, this risk is included in price of unit tests, low level tests, isn't it?

Best regards,
--
Mateusz ??oskot,

?

On 14 January 2014 12:05, Mateusz ?oskot <mateusz@...> wrote:
> 2) What about having both assertions, would that break the single
> assertion rule?
>
> self.assertTrue(item_exists)
> foo.xdep.get_item.assert_called_once_with(1)

> [...]

> 3) AFAIU, it's natural that unit tests are coupled with implementation of
> method/behaviour they are testing, foo.exists function in my case.

> [...]

I'm re-reading my own post and realising, aren't 2) and 3)
breaking the guideline [1] about writing tests to the interface,
not the implementation?

I think the original version of test_exists_ExistingItemId_ReturnTrue
I've got is written to the interface and mocks only bare minimum
of required dependencies.

Anyone could help me out to get in order with
terminology vs practice vs best practices here?

How much unit tests should know about implementation of what is being tested?

[1] "Stop Mocking, Start Testing" Augie Fackler and Nathaniel Manista at Google

Best regards,
--
Mateusz ?oskot,

?

Pity I didn't see this blog post first....
??

Because then I would have defined "Provocative Test that adds one new Behaviour" as one that provoked one new behaviour in the precedence order suggested by Uncle Bob.

On Mon, Dec 30, 2013 at 9:40 AM, John Carter <john.carter@...> wrote:

So I have being doing TDD for years now.

But I started noting I tend not to do it exactly "By The Book".

So as an exercise I tried going back to doing it...

* Very fine grained steps.

* Small change so Test fails - commit.

* Small change so Test passes - commit.

* Refactor so test passes - Commit.

As small as I could.

Oh, and just for the fun of it, I'm learning the D language, so I did it in D.

So here is the result, complete with a blow-by-blow Mercurial Repository of every step.

?Conclusions from the exercise.

1. Very fine grained steps resulted in fewer bugs and fewer backtracks.
2. Very fine grained steps resulted in better factored code.
3. Watching others do the same exercise led me to the attitude "adding more test cases is Bad". (Slower, hard to maintain tests, more lines of code)
4. Very fine grained steps slowed me down compared to my more usual somewhat loose style.
5. Thoughtfully adding Provocative test cases in the order of "What I need to Handle Next" is Very Very Good.
6. Replacing "Fine Grained Steps" with, whenever stuck, pull out a smaller? chunk and TDD that first (with Provocative test cases) works Very Very? Well.
   
   

Bottom Line?

In future I will be doing...

1. Thoughtfully choosing a minimal number of provocative test cases in the order that challenges a single new behaviour, ie. A test case that requires a single additional behaviour in the code.
2. I will take the largest steps I can where I can go from test fail to test pass in a single change.
3. Whenever I misjudged and take too large a step, I will factor out a? smaller function and test and develop that first via the same process.
4. Factoring out works best like this...
1. Revert to last passing state, leaving failing test commented out.
   
2. Copy pasta the code you wish to factor / enhance into a small function leaving original function untouched.
   
3. Get tests for new function to pass.
4. Wire new function into parent cleaning out original copy.
5. Get tests to pass.
6. Reenable commented out test... hopefully it passes now.
   

What do I mean by provocative test cases?

For example, when converting Roman numerals.

"I" is provocative.

"II" is provocative.

"III" is NOT provocative, it merely is extra Lines of Code (and hence a liability).

"IV" is provocative, but a bad choice for next test case as it adds two new behaviours at once.

"V" is provocative and adds a single new behaviour.

"IV" is now provocative and adds only a single new behaviour.

?

Conclusions about D.

?I love it.

?It is C / C++ with the worst stuff removed, and the best stuff stolen from it's competitors.

?If you are serious about industrial grade, high quality, defect free, efficient programming, you urgently need to be looking at D as your next language.

?Sigh! Unicode.

?That's not D's fault.

?D handles Unicode remarkably smoothly, but ye ancient 7bit ASCII is always going to be faster and conceptually easier than variable length code point UTF8.

?It took a bit of time and effort to get my head around D's template system (richer, safer, better than C++)

?It took even more time to get my head around Ranges, (again, richer, more flexible, safer than C++ iterators, more efficient than generators, coroutines or enumerators.)

?The choices D has made are rich in very powerful, very efficient implications.

Possibly the most striking is Compile Time Function Evaluation.

If it can be evaluated at compile time.... in D it will be.

And that is startling powerful.

--

John Carter
Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ?? ? ???
PO Box 1645 Christchurch
New Zealand

--

John Carter
Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ?? ? ???
PO Box 1645 Christchurch
New Zealand

---

This email, including any attachments, is only for the intended recipient. It is subject to copyright, is confidential and may be the subject of legal or other privilege, none of which is waived or lost by reason of this transmission.

If you are not an intended recipient, you may not use, disseminate, distribute or reproduce such email, any attachments, or any part thereof. If you have received a message in error, please notify the sender immediately and erase all copies of the message and any attachments.

Unfortunately, we cannot warrant that the email has not been altered or corrupted during transmission nor can we guarantee that any email or any attachments are free from computer viruses or other conditions which may damage or interfere with recipient data, hardware or software. The recipient relies upon its own procedures and assumes all risk of use and of opening any attachments.

---

So I have being doing TDD for years now.

But I started noting I tend not to do it exactly "By The Book".

So as an exercise I tried going back to doing it...

* Very fine grained steps.

* Small change so Test fails - commit.

* Small change so Test passes - commit.

* Refactor so test passes - Commit.

As small as I could.

Oh, and just for the fun of it, I'm learning the D language, so I did it in D.

So here is the result, complete with a blow-by-blow Mercurial Repository of every step.

?Conclusions from the exercise.

1. Very fine grained steps resulted in fewer bugs and fewer backtracks.
2. Very fine grained steps resulted in better factored code.
3. Watching others do the same exercise led me to the attitude "adding more test cases is Bad". (Slower, hard to maintain tests, more lines of code)
4. Very fine grained steps slowed me down compared to my more usual somewhat loose style.
5. Thoughtfully adding Provocative test cases in the order of "What I need to Handle Next" is Very Very Good.
6. Replacing "Fine Grained Steps" with, whenever stuck, pull out a smaller? chunk and TDD that first (with Provocative test cases) works Very Very? Well.
   
   

Bottom Line?

In future I will be doing...

1. Thoughtfully choosing a minimal number of provocative test cases in the order that challenges a single new behaviour, ie. A test case that requires a single additional behaviour in the code.
2. I will take the largest steps I can where I can go from test fail to test pass in a single change.
3. Whenever I misjudged and take too large a step, I will factor out a? smaller function and test and develop that first via the same process.
4. Factoring out works best like this...
1. Revert to last passing state, leaving failing test commented out.
   
2. Copy pasta the code you wish to factor / enhance into a small function leaving original function untouched.
   
3. Get tests for new function to pass.
4. Wire new function into parent cleaning out original copy.
5. Get tests to pass.
6. Reenable commented out test... hopefully it passes now.
   

What do I mean by provocative test cases?

For example, when converting Roman numerals.

"I" is provocative.

"II" is provocative.

"III" is NOT provocative, it merely is extra Lines of Code (and hence a liability).

"IV" is provocative, but a bad choice for next test case as it adds two new behaviours at once.

"V" is provocative and adds a single new behaviour.

"IV" is now provocative and adds only a single new behaviour.

?

Conclusions about D.

?I love it.

?It is C / C++ with the worst stuff removed, and the best stuff stolen from it's competitors.

?If you are serious about industrial grade, high quality, defect free, efficient programming, you urgently need to be looking at D as your next language.

?Sigh! Unicode.

?That's not D's fault.

?D handles Unicode remarkably smoothly, but ye ancient 7bit ASCII is always going to be faster and conceptually easier than variable length code point UTF8.

?It took a bit of time and effort to get my head around D's template system (richer, safer, better than C++)

?It took even more time to get my head around Ranges, (again, richer, more flexible, safer than C++ iterators, more efficient than generators, coroutines or enumerators.)

?The choices D has made are rich in very powerful, very efficient implications.

Possibly the most striking is Compile Time Function Evaluation.

If it can be evaluated at compile time.... in D it will be.

And that is startling powerful.

--

John Carter
Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ?? ? ???
PO Box 1645 Christchurch
New Zealand

?

So I have being doing TDD for years now.

But I started noting I tend not to do it exactly "By The Book".

So as an exercise I tried going back to doing it...

* Very fine grained steps.

* Small change so Test fails - commit.

* Small change so Test passes - commit.

* Refactor so test passes - Commit.

As small as I could.

Oh, and just for the fun of it, I'm learning the D language, so I did it in D.

So here is the result, complete with a blow-by-blow Mercurial Repository of every step.

?Conclusions from the exercise.

1. Very fine grained steps resulted in fewer bugs and fewer backtracks.
2. Very fine grained steps resulted in better factored code.
3. Watching others do the same exercise led me to the attitude "adding more test cases is Bad". (Slower, hard to maintain tests, more lines of code)
4. Very fine grained steps slowed me down compared to my more usual somewhat loose style.
5. Thoughtfully adding Provocative test cases in the order of "What I need to Handle Next" is Very Very Good.
6. Replacing "Fine Grained Steps" with, whenever stuck, pull out a smaller? chunk and TDD that first (with Provocative test cases) works Very Very? Well.
   
   

Bottom Line?

In future I will be doing...

1. Thoughtfully choosing a minimal number of provocative test cases in the order that challenges a single new behaviour, ie. A test case that requires a single additional behaviour in the code.
2. I will take the largest steps I can where I can go from test fail to test pass in a single change.
3. Whenever I misjudged and take too large a step, I will factor out a? smaller function and test and develop that first via the same process.
4. Factoring out works best like this...
1. Revert to last passing state, leaving failing test commented out.
   
2. Copy pasta the code you wish to factor / enhance into a small function leaving original function untouched.
   
3. Get tests for new function to pass.
4. Wire new function into parent cleaning out original copy.
5. Get tests to pass.
6. Reenable commented out test... hopefully it passes now.
   

What do I mean by provocative test cases?

For example, when converting Roman numerals.

"I" is provocative.

"II" is provocative.

"III" is NOT provocative, it merely is extra Lines of Code (and hence a liability).

"IV" is provocative, but a bad choice for next test case as it adds two new behaviours at once.

"V" is provocative and adds a single new behaviour.

"IV" is now provocative and adds only a single new behaviour.

?

Conclusions about D.

?I love it.

?It is C / C++ with the worst stuff removed, and the best stuff stolen from it's competitors.

?If you are serious about industrial grade, high quality, defect free, efficient programming, you urgently need to be looking at D as your next language.

?Sigh! Unicode.

?That's not D's fault.

?D handles Unicode remarkably smoothly, but ye ancient 7bit ASCII is always going to be faster and conceptually easier than variable length code point UTF8.

?It took a bit of time and effort to get my head around D's template system (richer, safer, better than C++)

?It took even more time to get my head around Ranges, (again, richer, more flexible, safer than C++ iterators, more efficient than generators, coroutines or enumerators.)

?The choices D has made are rich in very powerful, very efficient implications.

Possibly the most striking is Compile Time Function Evaluation.

If it can be evaluated at compile time.... in D it will be.

And that is startling powerful.

--

John Carter
Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ?? ? ???
PO Box 1645 Christchurch
New Zealand

---

This email, including any attachments, is only for the intended recipient. It is subject to copyright, is confidential and may be the subject of legal or other privilege, none of which is waived or lost by reason of this transmission.

If you are not an intended recipient, you may not use, disseminate, distribute or reproduce such email, any attachments, or any part thereof. If you have received a message in error, please notify the sender immediately and erase all copies of the message and any attachments.

Unfortunately, we cannot warrant that the email has not been altered or corrupted during transmission nor can we guarantee that any email or any attachments are free from computer viruses or other conditions which may damage or interfere with recipient data, hardware or software. The recipient relies upon its own procedures and assumes all risk of use and of opening any attachments.

---