Measurement and bias are often found together. Ignoring or denying this is a risky bet. But analysts always know how to draw useful learnings from this fact.
Analysts, whatever their field, must have the humility to admit that some of their measurements are skewed.
In digital analytics, this can arise from tagging, collection techniques, data consolidation, or for a whole host of other good reasons.
Measurement biases also exist in numerous fields of activity. Let’s take the example of Paco, William and Günther, who are going on holiday to Gran Bahia Principe Akumal (Yucatán Peninsula, Mexico).
Outbound trip from Madrid (MAD):
- Luggage weight, using a handheld travel scale (#1.1)
- Weighed again on the fixed scale at the airport (#1.2)
Return trip from Cancún (CUN):
- Luggage weight, using the same handheld travel scale (#2.1)
- Weighed again on the fixed scale at the airport (#2.2)
|Outbound from MAD (#1.1)||Outbound from MAD (#1.2)||Return from CUN (#2.1)||Return from CUN (#2.2)|
We notice a discrepancy of 0.8kg between the two outbound weights, and a discrepancy of 0.5kg between the two return weights. We can’t know whether the bias relates to one or more of the scales used (the handheld travel scale, the fixed scale at the Madrid airport, or the fixed scale at the Cancún airport).
Can we get relevant data, despite this identified (but unquantified) measurement bias?
Yes, there is quality information to be found here.
Let’s consider the weight discrepancies of one suitcase in relation to another, on the outbound trip (Günther has 4.7kg more than Paco) and on the return trip (Gunther has 1kg more than Paco), and the additional weight for each of them between the outbound and return trips:
- with the first measurement method (weights #1.1 and #2.1, standardised, handheld travel scale): Paco added 5.2kg, William 8.3kg and Günther 1.5kg
- with the second method (weights #1.2 and #2.2, not standardised, two different scales): Paco added 4.9kg, William 8kg and Günther 1.2kg
- There is a 0.3kg difference between the outbound discrepancies (0.8kg) and return trip discrepancies (0.5kg). Absolute values are therefore erroneous.
We can also analyse the measurement instruments:
- Fact: Discrepancy of 0.8kg between the handheld scales and fixed scales upon departure.
- Fact: Discrepancy of 0.5kg between the handheld scales and fixed scales upon return.
- You can infer a discrepancy of 0.3kg between the fixed scales upon departure and return… On condition, however, that you’re certain the handheld travel scale’s calibration (enabling standardisation) did not change between departure and return!!
- We can assume the handheld scale underestimates the weight, but it is impossible to know by how much. However, this will make it possible to counterbalance weights in the future…
But for other analyses, can’t bias lead to errors?
Yes, mainly when reasoning in terms of absolute values. The skewed nature of the measurement can pose a problem for certain analyses but has no effect on others.
For example: For the return trip, it is necessary to pack the suitcases in a way so as to remain within the luggage allowance of 23kg per suitcase.
- According to the first weighing, William needs to remove 1kg and place it in Paco’s suitcase, or share it between Paco and Günther.
- According to the second weighing, William needs to remove 1.5kg, but he can’t put anything in Günther’s suitcase and can only put 1kg in Paco’s. The remaining 0.5kg will be charged as a supplement.
How can I know if a measurement bias is acceptable, or not?
Most of the time, a bias that remains constant across time and space will allow for relevant analysis of comparative or relative values (discrepancies, proportions, etc.).
Once again, context is key for analysts: They must always define the format and content of their analyses based on the context.
So, as you can surely see, this issue applies to digital analytics in particular, as data collection is highly dependent on external factors, as is the choice of methods and techniques (tagging quality, first and foremost).
You tell us…
We’re interested in your comments and experiences!