Skip to content

A/B Testing

Availability: A/B testing is available on Professional and Agency plans.
Splitting traffic doesn’t tell you which variant works better: you need to measure what happens after the click. Turn on conversion tracking to compare each variant’s leads and sales, tag the variants to compare them in your own analytics, or ideally both. The link’s A/B settings show which of the two are on, in the Measuring this test box.

A/B testing sends visitors to different destinations from the same short link so you can compare performance. Shortn assigns each visitor to a variant and records the variant with every click (the A/B variant column of your clicks and click exports).

Splitting traffic alone doesn’t tell you which destination works better: a 50/50 split gets about 50/50 clicks whatever the pages do. To compare variants, measure what happens after the click:

  • With conversion tracking, Shortn counts the leads, sales and revenue each variant brings.
  • With variant tagging, your site’s own analytics can compare the variants (see below).

You can use both: conversions tell you which variant earns more, your analytics can tell you why, and it shows signs earlier, since visits come in long before sales. The two will never match exactly: Shortn counts clicks, your analytics counts sessions after cookie banners and ad blockers.

How visitors are assigned (no cookies)

Shortn splits visitors without cookies and without storing anything in their browser. It picks a variant from a one-way hash of the visitor’s IP address and browser, the link, and a random secret that changes every week. Nothing about the visitor is stored with the click except the variant’s letter.

This means:

  • A visitor usually keeps the same variant for about a week. When the weekly secret changes, they may get a different one.
  • A visitor can also get a different variant sooner if their network or browser changes, for example a phone moving from Wi-Fi to mobile data, or a browser update.
  • Two people on the same network with the same browser version get the same variant. Across all your visitors, the split still follows your percentages.

The A/B secret is separate from the one used to count unique visitors, which changes every day, and it’s never stored with a click.

If you need visitors to stay on one variant for longer, run the test on your own site, where your own consent setup applies.

Variants

A link can have up to four variants at a time. Each variant has a destination URL and a share of traffic.

  • Set an explicit percentage for each variant (they should add up to 100).
  • If you don’t set percentages, traffic is split equally.

The simplest setup compares two destinations — your current one and an alternative. Set the percentage for one; the rest goes to the other.

Variant letters

Each variant gets a letter (A, B, C…), and clicks are recorded under it. Every new test starts again at A; Shortn keeps each test’s clicks apart, so an earlier test’s A is never mixed up with a later test’s A.

During a test, a variant keeps its letter when you reorder variants or change their percentages, so its clicks always belong to the right destination. A new variant, or one whose URL you change, gets the next letter of that test, even if you removed another variant earlier. Until you save, a new variant shows + instead of a letter.

Because a changed address becomes a new variant, Shortn asks first: under the variant, choose Save as new variant or Cancel. Saving then waits 5 seconds with a Go back button; until it runs out nothing is saved, no letter is used and no visitor goes to the new address. If you leave or reload the page during those seconds, the change is saved.

Reordering variants doesn’t move any visitor to another variant. Changing the percentages does move some: when a variant’s share grows or shrinks, some visitors have to change side.

See the results

The link’s Overview shows an A/B test card for every link that has run a test. Pick a test to see it:

  • A running test lists its variants, including ones you removed, with their address, split and clicks.
  • An ended test shows only its winner.

Clicks follow the split, so on their own they don’t tell you which variant works better. When conversion tracking measures the test, each variant also shows its sales (or leads, while the test has no sales), its conversion rate and its revenue. Conversions count for the variant of the click that brought them.

Shortn names a leading variant only once the two best variants each have at least 100 clicks and 10 sales (or leads), and the leader is at least 95% likely to be better than the next best. Until then the card says there isn’t enough data yet, or that there’s no clear leader, so you don’t pick a winner on chance.

Tag variants for your analytics

In the A/B settings, the Measuring this test box shows whether conversion tracking and analytics tagging are on. Turn on Analytics tagging there, and visitors reach each variant with utm_content=variant-a, variant-b, and so on. Analytics tools such as Google Analytics, Plausible or Matomo already report on utm_content, so bounce rate, goals or purchases can be split by variant there. It’s a parameter in the address, not a cookie.

If the link, a variant’s address or the short link’s own address already sets utm_content, choose whether to add the variant to it (spring-sale-variant-b, the default) or replace it (variant-b). The setting shows the exact value visitors arrive with.

Tagging is off by default: turning it on changes the addresses visitors arrive at, so decide before the test starts if you can.

Review reminder

A test runs until you end it. To be reminded to look at the results, set a review reminder in the A/B settings: in 7, 14 or 30 days, or on a date you choose. On that day, at 09:00 in your timezone, everyone in the workspace gets a notification to review the test and pick a winner, and the link’s log records it.

  • The reminder is sent once. To be reminded again, set a new date.
  • It only reminds you: the test keeps running, and Shortn never picks a winner for you.
  • A paused link’s reminder waits until the link is live again.
  • Each member can turn these notifications off in their notification settings.

Ending a test

When you’ve decided, select the trophy next to the winning variant and confirm. Shortn then:

  • makes the winner’s URL the link’s destination, so every visitor goes there;
  • removes the variants and the review reminder;
  • records “A/B test ended” with the winner in the link’s log.

Past clicks keep their variant letters. If you start a new test on the same link later, it starts at A again, and its clicks are kept apart from the earlier test’s.

The winner’s destination goes through the same checks as any destination change, and the change counts toward your plan’s destination edits.

With the API, end a test with POST /api/v1/links/{code}/ab/winner and a body such as {"variant": "B"} (requires the links:write scope).

When a destination stops working

Shortn checks every destination of a link every 30 minutes, not only the main one: each variant is checked too, and a new or changed one right after you save it. If one stops working (it returns an error or can’t be reached):

  • its variant, and the A/B section in the link’s settings, shows a red warning, with the address and the error;
  • everyone in the workspace gets a notification once, and again when it works again;
  • the link’s log records both.

Visitors keep going to that destination until you fix or replace it: Shortn never moves them elsewhere on its own, which would also skew the test.

How A/B interacts with targeting

A/B is the last step in routing. If a targeting rule matches a visitor (by geography, language, device, OS, or browser), that destination wins and A/B doesn’t apply. A/B only splits the visitors who didn’t match any targeting rule.

If A/B isn’t enabled, those visitors go to the link’s default destination.

Last updated on