It's not the pretending so much as making the commit history tell an easy to understand (well organized) story about what changes were being made to the code.? The extreme opposite is a commit history full of commits with commit messages like "oops", "crap!", "forgot this bit in the previous commit", "dammit", "I hope this fixes things", etc.

I suppose if you work within an organization (and I have in the past) where merges to the main branch are required to be squash merges, then none of this storytelling, whether it's bad or good, matters much to anyone but oneself, but I prefer to write a well organized story even for myself -- the future self who has long since forgotten most of what he once knew about this portion of the code and will appreciate an easy to read/understand exposition of it rather than a stream of consciousness or play-by-play retelling of the actual events as they occurred.

Al

On Monday, June 14, 2021, 02:45:14 PM PDT, Ron Jeffries <ronjeffriesacm@...> wrote:

Hi Al,

sent from iPad, probably via Mars. Errors, if any, are not mine.

ronjeffries@... is a better address for me, maybe.

On Jun 14, 2021, at 9:22 AM, Al Chou via <hotfusionman=[email protected]> wrote:

?I can't remember or quickly find the blog post where I first saw it espoused to treat your private "feature" (in Git Flow terminology) as your own to rewrite commit history as you wish. I do it very frequently to better-organize the content of commits (e.g, pretend I wrote those tests at the same time as the production code and glom them together into the same new commit by squishing the two commits together, which has the advantage of making them an atomic package). I admit I only skimmed John's original post in the current thread, but I got the impression that's the kind of thing he was saying he uses the ability to rewrite VCS history for.

Yes, sounds similar. Why do you want to pretend those things? Do you go back and read the revised history or something? I wonder, because I always work on the tip of the spear, and only very rarely check out a prior version, generally to figure out when something changed. When I do feel the need to do that, I usually consider it a bit of a failure.

I'm about what the code is now?so I truly think wonder why folks care about the order of things in the past.?

Thanks,

R

I'm? with Ron on this. I want the history to be clean, for sure, but even more I want it to be correct. That means it should reflect what I did in the order that I did it. For example, maybe I got carried away in refactoring and made real (behavioral) changes, which broke some tests. If I haven't yet committed, I can fix that locally. But if I have, I'll leave it in and make a correction. Some of those corrections may be a bit embarrassing, but that will help me remember to do it right next time.

One thing I sometimes do, which is similar to what the OP suggests: When working on a script, which __only runs on a particular CI server _ (e.g. AppVeyor), it may take me several tries to get it right, especially if I don't fully understand how the CI server works. In that case, I'll eventually do a rebase, eliminating all my failed attempts. So sue me!

Charlie

On Mon, Jun 14, 2021 at 2:45 PM Ron Jeffries <ronjeffriesacm@...> wrote:

Hi Al,

sent from iPad, probably via Mars. Errors, if any, are not mine.

ronjeffries@... is a better address for me, maybe.

On Jun 14, 2021, at 9:22 AM, Al Chou via <hotfusionman=[email protected]> wrote:

?I can't remember or quickly find the blog post where I first saw it espoused to treat your private "feature" (in Git Flow terminology) as your own to rewrite commit history as you wish. I do it very frequently to better-organize the content of commits (e.g, pretend I wrote those tests at the same time as the production code and glom them together into the same new commit by squishing the two commits together, which has the advantage of making them an atomic package). I admit I only skimmed John's original post in the current thread, but I got the impression that's the kind of thing he was saying he uses the ability to rewrite VCS history for.

Yes, sounds similar. Why do you want to pretend those things? Do you go back and read the revised history or something? I wonder, because I always work on the tip of the spear, and only very rarely check out a prior version, generally to figure out when something changed. When I do feel the need to do that, I usually consider it a bit of a failure.

I'm about what the code is now?so I truly think wonder why folks care about the order of things in the past.?

Thanks,

R

Hi Al,

sent from iPad, probably via Mars. Errors, if any, are not mine.

ronjeffries@... is a better address for me, maybe.

On Jun 14, 2021, at 9:22 AM, Al Chou via <hotfusionman=[email protected]> wrote:

?I can't remember or quickly find the blog post where I first saw it espoused to treat your private "feature" (in Git Flow terminology) as your own to rewrite commit history as you wish. I do it very frequently to better-organize the content of commits (e.g, pretend I wrote those tests at the same time as the production code and glom them together into the same new commit by squishing the two commits together, which has the advantage of making them an atomic package). I admit I only skimmed John's original post in the current thread, but I got the impression that's the kind of thing he was saying he uses the ability to rewrite VCS history for.

Yes, sounds similar. Why do you want to pretend those things? Do you go back and read the revised history or something? I wonder, because I always work on the tip of the spear, and only very rarely check out a prior version, generally to figure out when something changed. When I do feel the need to do that, I usually consider it a bit of a failure.

I'm about what the code is now?so I truly think wonder why folks care about the order of things in the past.?

Thanks,

R

So there is a new tool on the block that permits new approaches to software development.

This is my first attempt at describing what can be done with it. Ultimately this will grow in a blog post and maybe? training course.

I'm working with the Mercurial Distributed Version Control System, which makes a lot of things a lot easier and simpler than git, but I believe nothing I say here cannot be emulated in git.

Here is the documentation, but it's not needed to understand what I'm saying.

The core idea is that changesets or commits to your version control system become mutable until you choose to publish them.

You can split them, join them, reorder them, rebase them on top of other commits.

Thus the "when" you do a change, becomes decoupled from "where" in the evolution sequence of the codebase. ie. You no longer have to make changes only at the end of an ever growing branch. You can make changes anywhere within your branch and at any time.

Declaring a change "fit for public consumption" is decoupled from "committing to version control".

This post is about what use you might make of? this decoupling.

Now keep in mind the definition of the word "Refactor". It means "Improving the code WITHOUT changing it's externally visible behaviour".

If you refactor the code and a test breaks.... you are either not doing a refactoring or your tests are not just testing behaviour, but are coupled to implementation details. More on that later.

Now in the game of TDD, you have a couple of moves you can make...

1. Refactor a test.
2. Extend a test. ie. Provide more test coverage of existing code
   
3. Refactor the code.
4. Change the Test to check for new behaviour.
   
5. Change the code behaviour..
6. Add observability. (Logging, tracepoints etc.)
7. Add inner checks (precondition asserts documenting my beliefs and assumptions about the code).
   

Refactoring a test or extending a test should NOT require a change of code. If it does, something is wrong. Either it was not a refactoring or extension, or the extension uncovered a preexisting bug in the code.

Number 6 is interesting and not usually mentioned in the context of TDD. It's sort of orthogonal to unit testing, in fact, unless it's a requirement like an audit trail, I'd explicitly strongly recommend you _don't_ unit test logging as it should NOT change the behaviour of the code whether it's turned on or off! However, as you will see later, it becomes a powerful additional tool in your armoury!

The traditional mantra of TDD is never write a line of code unless you have a breaking test.

Note that this imposes a timewise ordering on activities.

1. Write a test. Watch it break.
2. Implement the matching code. Watch it go green.
3. Refactor. Keep it green.

Now the point with changeset evolution is we don't care _when_ we do those things. We care about the order in the evolution of the codebase in which they occur.

For example, implementing a change in behaviour of the code should result in a test breaking (the tests are verifying the behaviour). If it doesn't our tests are insufficient. We should extend our tests.

Or conversely, if we write a test in anticipation of the next step, the implementation step, and it doesn't break, we again have something wrong.

Furthermore, most of us are not sufficiently lucky to always and only work with a fully TDD'd codebase with excellent coverage.

So when working with "legacy" code, there is a zeroth step... extend the test coverage.

So how much coverage is "enough"? Must we first get 100% coverage of everything we touch? What are we testing? Mostly that it just does whatever it does which we barely have a clue about.

The entire point of refactoring is to improve the internal qualities like readability, understandability and simplicity of the code.

Conversely a bundle of code ridden with technical debt is obtuse, and odds on you don't really have a clue what it does.

So how do you even start with a debt ridden legacy code?

I start with observability. I turn on logging, I add logging, I run the code to get a clue.

Commit!

I then add a "Hello World" unit test. Simplest dumbest stupidest test in the world. Starts the unit, shuts it down, cleans up and resources, nothing else.

Commit!

I then look at my coverage. It's lousy, almost zero.

But now between my logging and my coverage and my debugger, I can see an in, I can see where the happy path goes.

Add a test that goes? startup,? one step on the happy path, check it succeeded, teardown.

Commit!

But I can't make head or tail of what it's really doing, it's too complex.

So I sprinkle a few precondition assert checks to executably document what I believe about the system, run tests, whoops, one assumption was wrong, I've learnt something, remove or alter that check. Run, it's green.

Commit!

I can make the code simpler by some low risk, very "Tiny Step" refactorings that are "Obviously Correct & Better". Maybe early return pattern, maybe reduce scope of variables. Tiny tiny tiny step.

Commit!

Repeat several times. Commit! Commit! Commit!

Run up on target / system test. Oh Shit! It's broken! I did something stupid! Options?

Diff the logs before and after my changes. Aha! The behaviour diverged there! Oh dear, the logging is too sparse and coarse grained to tell me where.

MOVE BACK TO START OF THE BRANCH! BEFORE ALL TESTS! BEFORE ALL CHANGES!

Add more logging!

Run up and record log.

Rebase everything on top of the additional logging.

Run up and record log.

Diff the logs! Aha! Exactly there is the change!

Use "hg bisect" or "git bisect" to identify the changeset that broke it.

Hmm. Why didn't my unit test catch it? Look at coverage, oh dear, I don't cover that branch, or check that, oh dear, oh dear!

Extend the test! Does it catch it now! Yes!

MOVE BACK TO WHERE THE UNIT TESTS WAS ADDED! AND COMMIT THE TEST EXTENSION THERE!

Rebase everything. Go to the changeset that broke it.

Test now doesn't run.

Fix that changeset.

AMEND that changeset. So tests now run. Evolved / rebase everything on top of that.

So you see what we are growing here...

At the base of the branch....

* Only extensions to observability, only non-behaviour changing logging, you could drop all this stuff into the mainline right now. Zero risk.

* Followed by Only additions of, or extensions to unit tests, so you could drop all this stuff into the mainline right now. Zero risk.

* Followed by an ever growing pile of tiny tiny tiny PURE refactorings.

* Followed by a cluster of small neat changes in behaviour.

The real time WHEN we add logging or test cases or refactorings, is decoupled from WHERE in the branch we insert them.

IMPORTANT RULE! At _every_ commit, everything always compiles, links, and all tests run successfully. (Except maybe a Work In Progress changeset at the very end)

If at any stage the tip is broken, you can bisect to the breaking changeset and fix and amend and evolve.

And so you go, improving the code, extending the test coverage, improving the observability.

At some point, you reach the objective.... You now understand the code. You can see where to add the new feature / change of behaviour. It's clean and easy to add..

So you add a test at the end of your branch that tests for the existence of the new behaviour. It breaks. Add the code to implement. It passes.

Commit.

Now you need to add more, you start to do that...

Add test.... it breaks. Start to add code. Damn. I need to clean up more. Commit as a "Work In Progress".

MOVE BACK BEFORE THE BEHAVIOUR CHANGE.

Clean up. Commit! Evolve!

?

MOVE FORWARD TO TIP.

Add code, it's a small change now. The test passes. Amend the changeset.

On looking back at your implementation of the first feature, you spot an improvement.

MOVE BACK.

Refactor. Commit! Evolve!

Hmm. That first try was so hideous, I'm embarrassed it exists. No problem. I can fold the original implementation and my refactoring into one. Red face gone.

Keep doing everything above until....

1. The code is well factored, understandable testable and tested clean code running in production.
   
2. You completely understand the code as proven by...
1. Executable precondition checks documenting your assumptions.
2. Well designed unit tests that read as "executable documentation" of the subsystems behaviour.
3. Observable behaviour both under test and in production.
   
3. All required behaviour is implemented and tested in a clean manner.

But oh dear! All this is taking longer than a day. The rest of the Herd of Cats is pouring in code into the mainline at one man day's worth of code per cat per day.... you are heading for a Big Bang integration nightmare!

No problem. An hour or so before home time, get to a clean point, pull the mainline, move to a point already proven by the CI system....

AND REBASE YOUR ENTIRE BRANCH ON THAT POINT!

Go to the tip of your branch, does it compile and run? No. Bisect, fix, amend, evolve until it does.

Push to mercurial or git server and go home. Your stuff is backed up, the CI system will wake up and prove ( or otherwise) your branch.

Come to work in the morning, fix and amend anything the CI system complained about.

Carry on until you are done, reviewed, CI's happy. Rebase one last time and drop it into the mainline and push.

So what have we gained?

1. A powerful new debug tool. Growing and Diffing the logs. Especially effective in multithreaded apps. (ps: Also works on logs from unit tests and/or logs from full system tests!)
   
2. Mitigation of the risks of Big Bang integrations.
3. Mitigation of the risks of large refactorings.
4. An effective strategy for learning, covering, refactoring and changing legacy code.
5. Executable evidence of our understanding of the code, and executable documentation of what we have learnt.
6. A strategy to simplify review. Each changeset is tiny tiny tiny and obviously correct. You can review by changeset, or end to end. Your choice.
7. A clear separation between observability changes, test extensions, true refactorings and behaviour changes.
8. A gradation of risk shading from zero to some risk, allowing you to focus your test, review, debug efforts where they count.
9. A rapid strategy (bisection) for rapidly finding and fixing breaking changes.
10. A system to avoid interleaving changes from multiple programmers, to avoid broken mainlines, and to make it easy to pinpoint breaking changes once the branch is published.
    

--

John Carter
Tait Electronics? ? ? ? ? ? ? ? ?? ? ???
New Zealand

---

This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

---

So there is a new tool on the block that permits new approaches to software development.

This is my first attempt at describing what can be done with it. Ultimately this will grow in a blog post and maybe? training course.

I'm working with the Mercurial Distributed Version Control System, which makes a lot of things a lot easier and simpler than git, but I believe nothing I say here cannot be emulated in git.

Here is the documentation, but it's not needed to understand what I'm saying.

The core idea is that changesets or commits to your version control system become mutable until you choose to publish them.

You can split them, join them, reorder them, rebase them on top of other commits.

Thus the "when" you do a change, becomes decoupled from "where" in the evolution sequence of the codebase. ie. You no longer have to make changes only at the end of an ever growing branch. You can make changes anywhere within your branch and at any time.

Declaring a change "fit for public consumption" is decoupled from "committing to version control".

This post is about what use you might make of? this decoupling.

Now keep in mind the definition of the word "Refactor". It means "Improving the code WITHOUT changing it's externally visible behaviour".

If you refactor the code and a test breaks.... you are either not doing a refactoring or your tests are not just testing behaviour, but are coupled to implementation details. More on that later.

Now in the game of TDD, you have a couple of moves you can make...

1. Refactor a test.
2. Extend a test. ie. Provide more test coverage of existing code
   
3. Refactor the code.
4. Change the Test to check for new behaviour.
   
5. Change the code behaviour..
6. Add observability. (Logging, tracepoints etc.)
7. Add inner checks (precondition asserts documenting my beliefs and assumptions about the code).
   

Refactoring a test or extending a test should NOT require a change of code. If it does, something is wrong. Either it was not a refactoring or extension, or the extension uncovered a preexisting bug in the code.

Number 6 is interesting and not usually mentioned in the context of TDD. It's sort of orthogonal to unit testing, in fact, unless it's a requirement like an audit trail, I'd explicitly strongly recommend you _don't_ unit test logging as it should NOT change the behaviour of the code whether it's turned on or off! However, as you will see later, it becomes a powerful additional tool in your armoury!

The traditional mantra of TDD is never write a line of code unless you have a breaking test.

Note that this imposes a timewise ordering on activities.

1. Write a test. Watch it break.
2. Implement the matching code. Watch it go green.
3. Refactor. Keep it green.

Now the point with changeset evolution is we don't care _when_ we do those things. We care about the order in the evolution of the codebase in which they occur.

For example, implementing a change in behaviour of the code should result in a test breaking (the tests are verifying the behaviour). If it doesn't our tests are insufficient. We should extend our tests.

Or conversely, if we write a test in anticipation of the next step, the implementation step, and it doesn't break, we again have something wrong.

Furthermore, most of us are not sufficiently lucky to always and only work with a fully TDD'd codebase with excellent coverage.

So when working with "legacy" code, there is a zeroth step... extend the test coverage.

So how much coverage is "enough"? Must we first get 100% coverage of everything we touch? What are we testing? Mostly that it just does whatever it does which we barely have a clue about.

The entire point of refactoring is to improve the internal qualities like readability, understandability and simplicity of the code.

Conversely a bundle of code ridden with technical debt is obtuse, and odds on you don't really have a clue what it does.

So how do you even start with a debt ridden legacy code?

I start with observability. I turn on logging, I add logging, I run the code to get a clue.

Commit!

I then add a "Hello World" unit test. Simplest dumbest stupidest test in the world. Starts the unit, shuts it down, cleans up and resources, nothing else.

Commit!

I then look at my coverage. It's lousy, almost zero.

But now between my logging and my coverage and my debugger, I can see an in, I can see where the happy path goes.

Add a test that goes? startup,? one step on the happy path, check it succeeded, teardown.

Commit!

But I can't make head or tail of what it's really doing, it's too complex.

So I sprinkle a few precondition assert checks to executably document what I believe about the system, run tests, whoops, one assumption was wrong, I've learnt something, remove or alter that check. Run, it's green.

Commit!

I can make the code simpler by some low risk, very "Tiny Step" refactorings that are "Obviously Correct & Better". Maybe early return pattern, maybe reduce scope of variables. Tiny tiny tiny step.

Commit!

Repeat several times. Commit! Commit! Commit!

Run up on target / system test. Oh Shit! It's broken! I did something stupid! Options?

Diff the logs before and after my changes. Aha! The behaviour diverged there! Oh dear, the logging is too sparse and coarse grained to tell me where.

MOVE BACK TO START OF THE BRANCH! BEFORE ALL TESTS! BEFORE ALL CHANGES!

Add more logging!

Run up and record log.

Rebase everything on top of the additional logging.

Run up and record log.

Diff the logs! Aha! Exactly there is the change!

Use "hg bisect" or "git bisect" to identify the changeset that broke it.

Hmm. Why didn't my unit test catch it? Look at coverage, oh dear, I don't cover that branch, or check that, oh dear, oh dear!

Extend the test! Does it catch it now! Yes!

MOVE BACK TO WHERE THE UNIT TESTS WAS ADDED! AND COMMIT THE TEST EXTENSION THERE!

Rebase everything. Go to the changeset that broke it.

Test now doesn't run.

Fix that changeset.

AMEND that changeset. So tests now run. Evolved / rebase everything on top of that.

So you see what we are growing here...

At the base of the branch....

* Only extensions to observability, only non-behaviour changing logging, you could drop all this stuff into the mainline right now. Zero risk.

* Followed by Only additions of, or extensions to unit tests, so you could drop all this stuff into the mainline right now. Zero risk.

* Followed by an ever growing pile of tiny tiny tiny PURE refactorings.

* Followed by a cluster of small neat changes in behaviour.

The real time WHEN we add logging or test cases or refactorings, is decoupled from WHERE in the branch we insert them.

IMPORTANT RULE! At _every_ commit, everything always compiles, links, and all tests run successfully. (Except maybe a Work In Progress changeset at the very end)

If at any stage the tip is broken, you can bisect to the breaking changeset and fix and amend and evolve.

And so you go, improving the code, extending the test coverage, improving the observability.

At some point, you reach the objective.... You now understand the code. You can see where to add the new feature / change of behaviour. It's clean and easy to add..

So you add a test at the end of your branch that tests for the existence of the new behaviour. It breaks. Add the code to implement. It passes.

Commit.

Now you need to add more, you start to do that...

Add test.... it breaks. Start to add code. Damn. I need to clean up more. Commit as a "Work In Progress".

MOVE BACK BEFORE THE BEHAVIOUR CHANGE.

Clean up. Commit! Evolve!

?

MOVE FORWARD TO TIP.

Add code, it's a small change now. The test passes. Amend the changeset.

On looking back at your implementation of the first feature, you spot an improvement.

MOVE BACK.

Refactor. Commit! Evolve!

Hmm. That first try was so hideous, I'm embarrassed it exists. No problem. I can fold the original implementation and my refactoring into one. Red face gone.

Keep doing everything above until....

1. The code is well factored, understandable testable and tested clean code running in production.
   
2. You completely understand the code as proven by...
1. Executable precondition checks documenting your assumptions.
2. Well designed unit tests that read as "executable documentation" of the subsystems behaviour.
3. Observable behaviour both under test and in production.
   
3. All required behaviour is implemented and tested in a clean manner.

But oh dear! All this is taking longer than a day. The rest of the Herd of Cats is pouring in code into the mainline at one man day's worth of code per cat per day.... you are heading for a Big Bang integration nightmare!

No problem. An hour or so before home time, get to a clean point, pull the mainline, move to a point already proven by the CI system....

AND REBASE YOUR ENTIRE BRANCH ON THAT POINT!

Go to the tip of your branch, does it compile and run? No. Bisect, fix, amend, evolve until it does.

Push to mercurial or git server and go home. Your stuff is backed up, the CI system will wake up and prove ( or otherwise) your branch.

Come to work in the morning, fix and amend anything the CI system complained about.

Carry on until you are done, reviewed, CI's happy. Rebase one last time and drop it into the mainline and push.

So what have we gained?

1. A powerful new debug tool. Growing and Diffing the logs. Especially effective in multithreaded apps. (ps: Also works on logs from unit tests and/or logs from full system tests!)
   
2. Mitigation of the risks of Big Bang integrations.
3. Mitigation of the risks of large refactorings.
4. An effective strategy for learning, covering, refactoring and changing legacy code.
5. Executable evidence of our understanding of the code, and executable documentation of what we have learnt.
6. A strategy to simplify review. Each changeset is tiny tiny tiny and obviously correct. You can review by changeset, or end to end. Your choice.
7. A clear separation between observability changes, test extensions, true refactorings and behaviour changes.
8. A gradation of risk shading from zero to some risk, allowing you to focus your test, review, debug efforts where they count.
9. A rapid strategy (bisection) for rapidly finding and fixing breaking changes.
10. A system to avoid interleaving changes from multiple programmers, to avoid broken mainlines, and to make it easy to pinpoint breaking changes once the branch is published.
    

--

John Carter
Tait Electronics? ? ? ? ? ? ? ? ?? ? ???
New Zealand

---

This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

---

John,

I confess I'm not seeing the shine on this. I freely grant that I'm not doing any version control with a team of people, so that could be the whole reason.

However, it seems to me that if the team always works from the front of the version control queue, always pulling from and pushing to the head, this editing of the past has no discernible effect: head is always as good as we can make it.

What am I missing?

Thanks,

R

On Jun 10, 2021, at 10:12 PM, John Carter via <john.carter@...> wrote:

So there is a new tool on the block that permits new approaches to software development.

This is my first attempt at describing what can be done with it. Ultimately this will grow in a blog post and maybe? training course.

Ron Jeffries

An Agile method is a lens. The work is done under the lens, not by the lens.