Einstein

Measurement and bias are often found together. Ignoring or denying this is a risky bet. But analysts always know how to draw useful learnings from this fact.

Analysts, whatever their field, must have the humility to admit that some of their measurements are skewed.

In digital analytics, this can arise from tagging, collection techniques, data consolidation, or for a whole host of other good reasons.

Measurement biases also exist in numerous fields of activity. Let’s take the example of Paco, William and Günther, who are going on holiday to Gran Bahia Principe Akumal (Yucatán Peninsula, Mexico).

Outbound trip from Madrid (MAD):

  • Luggage weight, using a handheld travel scale (#1.1)
  • Weighed again on the fixed scale at the airport (#1.2)

Return trip from Cancún (CUN):

  • Luggage weight, using the same handheld travel scale (#2.1)
  • Weighed again on the fixed scale at the airport (#2.2)
Outbound from MAD (#1.1) Outbound from MAD (#1.2) Return from CUN (#2.1) Return from CUN (#2.2)
Paco 16.3kg 17.1kg 21.5kg 22.0kg
William 15.7kg 16.5kg 24.0kg 24.5kg
Günther 21.0kg 21.8kg 22.5kg 23.0kg

 

We notice a discrepancy of 0.8kg between the two outbound weights, and a discrepancy of 0.5kg between the two return weights. We can’t know whether the bias relates to one or more of the scales used (the handheld travel scale, the fixed scale at the Madrid airport, or the fixed scale at the Cancún airport).

ninja interogation

Can we get relevant data, despite this identified (but unquantified) measurement bias?

 

Yes, there is quality information to be found here.

Let’s consider the weight discrepancies of one suitcase in relation to another, on the outbound trip (Günther has 4.7kg more than Paco) and on the return trip (Gunther has 1kg more than Paco), and the additional weight for each of them between the outbound and return trips:

  • with the first measurement method (weights #1.1 and #2.1, standardised, handheld travel scale): Paco added 5.2kg, William 8.3kg and Günther 1.5kg
  • with the second method (weights #1.2 and #2.2, not standardised, two different scales): Paco added 4.9kg, William 8kg and Günther 1.2kg
  • There is a 0.3kg difference between the outbound discrepancies (0.8kg) and return trip discrepancies (0.5kg). Absolute values are therefore erroneous.

We can also analyse the measurement instruments:

  • Fact: Discrepancy of 0.8kg between the handheld scales and fixed scales upon departure.
  • Fact: Discrepancy of 0.5kg between the handheld scales and fixed scales upon return.
  • You can infer a discrepancy of 0.3kg between the fixed scales upon departure and return… On condition, however, that you’re certain the handheld travel scale’s calibration (enabling standardisation) did not change between departure and return!!
  • We can assume the handheld scale underestimates the weight, but it is impossible to know by how much. However, this will make it possible to counterbalance weights in the future…

measures relativity analytics

ninja interogation

But for other analyses, can’t bias lead to errors?

 

Yes, mainly when reasoning in terms of absolute values. The skewed nature of the measurement can pose a problem for certain analyses but has no effect on others.

For example: For the return trip, it is necessary to pack the suitcases in a way so as to remain within the luggage allowance of 23kg per suitcase.

  • According to the first weighing, William needs to remove 1kg and place it in Paco’s suitcase, or share it between Paco and Günther.
  • According to the second weighing, William needs to remove 1.5kg, but he can’t put anything in Günther’s suitcase and can only put 1kg in Paco’s. The remaining 0.5kg will be charged as a supplement.

ninja interogation

How can I know if a measurement bias is acceptable, or not?

 

Most of the time, a bias that remains constant across time and space will allow for relevant analysis of comparative or relative values (discrepancies, proportions, etc.).

Once again, context is key for analysts: They must always define the format and content of their analyses based on the context.

So, as you can surely see, this issue applies to digital analytics in particular, as data collection is highly dependent on external factors, as is the choice of methods and techniques (tagging quality, first and foremost).

 

You tell us…

We’re interested in your comments and experiences!

 

Author

Knowledge Manager After On-the-job and Off-the-job training in purchasing and management at Carrefour, and sales training at Procter & Gamble, JM evolved in the mass retail sector in top management positions for large hypermarket, central purchasing and logistics groups, with an expatriate experience in Africa as a Central Director. In late 1995, JM created an Internet start-up company and after three years (late 1998) he joined Alain Llorens and the AT Internet team where he took up his position in sales, and was also at the heart of the pioneering adventure in Web analytics. At 55 and after almost 13 years seniority in the company, JM has been Knowledge Manager since 2009.

Comments are closed.