Keeping a large testing programme on the right side of the line
The more money a testing programme makes, the more pressure there is to make more, and that's when your ethics need to be written down and enforced.
A small testing programme can get by on the good intentions of the person running it. A large one can’t, and pretending it can is how good programmes drift into doing things nobody meant to do. Governance is the set of rules that still hold when the person who wrote them isn’t in the room, and at scale those are the rules that count.
The drift is gradual, which is what makes it dangerous. A programme starts with a clear line: we won’t use fake scarcity, we won’t hide the unsubscribe, we won’t mislead. Then someone runs a test a little closer to the line, it wins, the win gets celebrated, and the line moves a little. Do that for two years and the programme no longer remembers where the line was, because every step felt reasonable at the time. Governance stops the line moving without anyone deciding it should.
The mechanics are less exciting than the principle. Write down what you won’t do as a list of specific things rather than a vague value, and check every test against it before it runs. Keep a record of decisions, so when someone asks why a test was allowed there’s an answer other than a shrug. And make saying no someone’s job, because when it’s everyone’s job nobody does it, and a rule nobody has the authority to enforce is only a suggestion.
Ethics at scale is also about who gets to decide, and it shouldn’t be the person who benefits from the decision. The team that runs a test is the team that feels the pressure to ship the winner, and that pressure is real. A sound structure separates the people who run the tests from the people who judge whether they should have run at all, and gives the judges the standing to say no to someone senior without risking their job. That separation is what stops the pressure from writing the rules.
Governance can also turn into paperwork: documents filled in, filed and never read, there to make the programme look responsible while nothing changes. You can spot it when the rules are written too vaguely to break and nobody can name a test that was refused. A governance model that never says no is decoration, and a programme with decoration is more dangerous than one that admits it has no governance, because at least the second one knows it’s relying on luck. Write the rules plainly, give someone the authority to enforce them, and keep an accurate record, and the line will stay where you drew it.
Put it to work
Write down, as a specific list, the tests you will not run. Check your last three tests against it, and note any that would have failed.
There is no right answer. Write yours first, then compare.
Sources
Shoshin is written by Another Web is Possible. Lessons are licensed CC BY-SA 4.0.Our promise.