Effects of small changes

By: Mat Moore

Tags:

  • graphs
  • user experience

A popular feature on our sites is something we call Encounters. It’s a matching game which lets you find users who want to meet you and vice versa. This can be accessed from various parts of our platform but a significant portion of the traffic comes from the link on the mobile homepage. In a recent update we decided this feature was now well-enough known with our members that we no longer needed to overlay a little “new” flag on the encounters icon.

image

Screenshot showing the encounters icon on the singles365.com homepage with the “new” flag present.

This was a minor part of a much larger release so we weren’t actively monitoring the impact as we performed our live tests. The deploy went smoothly, the new features we’d introduced were working well so, come the end of the day, we went home.

When we started work the next day, we noticed there had been a drop in traffic for Encounters. As this feature was so popular with our members, we quickly held a group investigation to work out what was wrong and what to do. Our ops team confirmed there was nothing broken and live testing re-affirmed this. Then someone mentioned that maybe it was to do with us removing the “new” label. Having ruled everything else out, we figured we’d check it out and see if that really could be to blame. We quickly made a pull request to add the label back in, ran it through QA then pushed it live. This time we were actively monitoring its impact and what we found was pretty surprising. As you can see from the graph below, traffic pretty much doubled instantly. All from one tiny little label.

image

Chart showing normalised traffic for 1 hour before and after adding the “new” label.

One of the best things about this department is that everyone’s input is listened to and we all get a chance to contribute, even if we’re not a specialist. This graph triggered a busy conversation and suddenly there were all sorts of theories and new ideas flying around. What did this mean? How could we use this information to help drive user engagement? Can we use this information to make other features easier to use? We’re still looking into it and I’m sure it’s going to result in some cool stuff but for me the most exciting part is that such a small change can be so powerful and the amount of optimisation this will inspire us to work on.


About the Author

Mat Moore