Keyboard Shortcuts
ctrl + shift + ? :
Show all keyboard shortcuts
ctrl + g :
Navigate to a group
ctrl + shift + f :
Find
ctrl + / :
Quick actions
esc to dismiss
Likes
Search
Re: Changeset Evolution as an aid to Test Driven Development.
|
On Wed, Jun 16, 2021 at 10:52 AM Ron Jeffries <ronjeffriesacm@...> wrote:
The poster child environment for this trick is embedded / real time / multithreaded. The further you're away from that the less benefit it provides. On an embedded target events are firing from the many different hardware subsystems at whatever time and in whatever order the hardware pleases and is handled in whatever order the scheduler pleases. Defect free code for _a_ module will cope with that, and sequence it's behaviour into something sensible and orderly. ie. Logs for _all_ modules mixed together will be pretty chaotic.? For a well behaved module, usually pretty orderly. The most difficult subtle horrible bugs in a multithreaded / real time environment arise from racy behaviour. ie. The code hasn't been written robust against different ordering or timing of events. All else being the same, refactorings of all and every sort will alter the ordering and timing of events, which in defect free code will make no difference... So if a refactoring, somewhere in your pile of refactorings, mixed up with a pile of behaviour changes, did something/triggered something nasty. Your pain is immeasurable. If you had adequate logging you would at least see where the behaviour changed, but you haven't. So you add logging, as per tradition at the end of your branch, and yes, you can see something changed somewhere in your pile. You have no clue as to which, you have to start as if you? know nothing. Which is where this technique becomes a super power.... go back to the start of the branch... add logging for the module of interest, record the real time behaviour. Rebase pile of refactorings on top, record new real time behaviour.... if true refactoring, nothing interesting will have changed. If something noxious and subtle is happening (perhaps even due to a preexisting bug), you can see exactly where the behaviour changed. Often that is enough to narrow down the exact changeset. Otherwise bisect to find it. Quite possibly that refactoring merely _exposed_ racy behaviour rather than introduced it. In which case? extend unit test to catch it, (insert at end of unit test region of branch), fix bug, check to see it did and then decide where you want to drop the fix. As I said, the poster child is embedded / real time / multithreaded, but value from this trick can even be gained within unit tests when the class under test is too complex to understand, and a refactoring broke something that the unit test coverage didn't catch. It tells you where precisely coverage is needed. But hey, why refactor if everything is easy and simple to understand? It's the gnarliest and worst modules with the most debt that most need refactoring.
If you are green fields and the code has never been released.. you get nothing. If the code has man years of testing and man decades or centuries of use in the field.....? you need fairly strong evidence that you are making things better, rather than just changing things and maybe making it worse. So extending the tests _after_ behavioural changes loses that valuable oracle.? Remember tests are code hence tests have bugs, tests test the code and the code tests the tests. So expanding tests on known working code, and adding precondition asserts, provides a strong oracle to prove that the tests are correct and working. Which as you grow the tests, they turn around and provide a strong oracle to prove your refactorings are correct and are indeed pure refactorings. ?
Breaks it down into small manageable integrations... and instead of finding later you are conflicting with weeks worth of work with another cat... You find out after only a days worth of conflicts, know to walk over and work _with_ the other cat. Yup, shouldn't ever happen good team work and all that.... blah blah, but shit and schedule pressure happens. Hanging out on a branch whilst man years is going into the mainline is going to be bad no matter what, but this trick decreases the urgency to close slightly.
Interposing behavioural changes with behavioural changes from other cats makes review and defect isolation hard. So if you're doing this mix of things, being able to drop the zero / low risk items first until you can build coverage and confidence. In the realm of excellent unit test coverage not so much of a problem, in browner fields, deeply embedded, multi-threaded, more of a problem.
Review. Makes it much much easier. Whether by yourself, or reviewer or pair. Bundle twenty or so refactorings... and it becomes damn hard to reason about. I have great confidence in myself in making correct refactorings.... so can blithely pile change upon change upon change. Alas, history shows that confidence is woefully unfounded.
Your unit tests are what give you confidence bravely refactor, knowing you aren't causing regressions. Sadly, looking at coverage analysis I know myself and all other cats are way overly optimistic about what unit tests do in fact test. Again, it's hard to assert negatives. It's easy to assert behaviour, but hard to assert the absence thereof. (Mutation testing sounds like a very good thing, anyone know of a good C/C++ mutation testing framework?) If a branch you have just refactored is showing red in coverage analysis.... your test? shouldn't be giving _any_ confidence about the change. Doesn't mean you need to abandon the change, just means you have to go back and brace it first. This Communication is Confidential. We only send and receive email on the basis of the terms set out at |
to navigate to use esc to dismiss