Pay

Change one thing, wait a fortnight, read the result

A price experiment that changes three variables at once teaches you nothing at four weeks' cost.

By 5 min readPay

Guides on Pay: Pricing, from first principles to the annual review, The ceiling is hours, and it arrives sooner than people expect, What rating work actually pays

You cannot run a proper price test with one listing and a dozen enquiries a fortnight, but you can run a legible one: change one variable, hold it two weeks or more, and read one metric chosen in advance.

That is weaker than an experiment, and enormously better than changing the price, the description and the tier structure on the same evening and then wondering.

Why the usual approach teaches nothing

The instinctive move after a quiet week is to fix everything at once. New price, rewritten first line, an extra tier, a different turnaround.

If bookings improve you do not know which change did it, so you cannot do it again. If they get worse you do not know which change to undo, so you undo all of them and are back where you started having spent a fortnight.

There is a second problem specific to low volume. Booking counts of three and five are not different numbers in any meaningful sense. Random variation at that scale swamps almost any effect you could produce, which is why the metric matters more than the discipline.

The metric: enquiry to booking, not revenue

Revenue is the wrong signal at low volume for two reasons. It is dominated by whether one large job happened to land, and it conflates traffic with conversion.

Enquiry-to-booking rate separates those. It answers the actual question a price change asks - of the people who were interested enough to write, how many said yes - and it is unaffected by whether the platform happened to feature you that week.

Signal What moves it Useful for a price test
Revenue Traffic, price, mix, luck No
Bookings per week Traffic and conversion together Weakly
Enquiry-to-booking rate Conversion, mostly price and clarity Yes
Enquiries per week Visibility and the listing's front line Only for visible-price changes

The pair worth watching together is the last two. A price rise that leaves conversion flat but halves enquiries is a visibility problem, because the price is being read in the grid before anyone writes. A price rise that leaves enquiries flat but halves conversion is a genuine price problem, and those are different repairs.

This means you have to count enquiries, including the ones that went nowhere. That is the whole reason tracking the enquiries that did not convert is worth the two minutes a week it costs - without a denominator you have no rate, and without a rate you have anecdotes.

How long to hold

A fortnight is the floor and it is often not enough. The honest rule is to hold until you have counted at least fifteen enquiries under the new condition, however long that takes, and to accept that at three enquiries a week that is five weeks.

Holding is the hard part, because the first week of any change is usually bad in a way that has nothing to do with the change. Listings reshuffle, buyers who saw the old price take a while to cycle through, and your own confidence dips in a way that shows up in how you write replies. What to do when bookings stop after a rise is the longer version of that specific panic, and the short version is that a fortnight of silence is usually a fortnight.

What to test, in order

Test the things with the largest expected effect first, because you only get a few tests a quarter.

  1. Whether the price is visible at all. Larger effect than the price itself, usually, and it moves enquiry volume rather than conversion.
  2. The middle tier's number. Move it by fifteen to twenty per cent. Smaller moves are unreadable at your volume.
  3. The presence of a top tier. Adding one changes how the middle reads without changing the middle's price.
  4. Turnaround as a priced feature. Splitting fast and normal delivery is a scope change disguised as a price change, and it usually reads well.
  5. The first line of the listing. Not a price change, but it competes for the same enquiries and will contaminate any test you run alongside it.

Do not test tier names, colour, ordering or anything else that cannot plausibly move a number by ten per cent. You do not have the volume to detect it and you will spend a test slot finding out.

Not annoying anyone

The etiquette question is real, and it has a simple resolution: buyers do not track your prices, but repeat buyers do.

Someone who booked at 35 and returns to find 45 will notice, and the moment they notice is the moment your policy needs to already exist rather than be improvised - which is why deciding the grandfathering rule before the first regular asks belongs in the preparation for a test, not the aftermath.

Beyond that, keep the cadence slow. One change a month is fine, one a week reads as instability, and a listing whose price visibly moves in both directions invites people to wait for the low point. The larger-scale evidence cuts the same way: in a 28-month field experiment with over 50,000 customers, Anderson and Simester (2010) found that buyers who later saw the same retailer selling an item for less made fewer subsequent purchases, and the effect was largest among its most valuable customers. Never advertise the test. "Trying out a new price this month" converts a price into a negotiation.

Buyers commissioning a person tend to describe the decision as a risk judgement more than a price comparison, and the account of how that choice gets made is a useful corrective when a test comes back flat: you may be moving the wrong variable entirely. The reason to be careful about small effects is the same reason careful measurement is a subject of its own - the conventions that make repeated figures comparable exist because uncontrolled measurement produces confident nonsense, and a five-enquiry price test is exactly that. And the floor your test runs against is not static: what an automated score can reliably establish keeps expanding, so a result from last year is not evidence about this one.

If your prices are set inside a platform's tier structure, the practical constraint is how often listings can change and how quickly the grid reflects it - worth confirming against the current judge listing setup before you start a clock you cannot actually run.

Read next

Full archive