An article on Aligned Genetics website [1] prompted this blog as it raises a very real and broader issue observed with so many technologies. “…there is a growing perception that “using a machine guarantees accuracy.” [1]
As a scientific instrument supplier for many decades, ATA Scientific has witnessed the technological progress of many products, observing the trend to simplify operation to enable a broader userbase, taking what was the realm of a few committed academics hidden in the basement of a university to ubiquity in industry. This sparks the question is the simplifying of operation dumbing down science?
As an aside, it is frightening to see a technology I used to supply as the latest cutting edge, frontiers of science system, relegated to a public museum (and my son pointing it out to me).
Are my results real?
This is a multifaceted question. From an instrument perspective – the answer is… well it depends! In many systems it is a function of the best we can get, an estimate, an accepted limitation, and it is these core limitations that are of concern, especially if the operator assumes the result to be absolute. Concerned? Well let’s explore the premise behind this statement. Consider it is the 1980’s and you have a brand spanking new Malvern laser diffraction system to measure particle size – you throw a sample into the dispersion bath and say go… out spits the result – happy days! But wait – the devil is in the detail. See, since the 1970’s they all used the Fraunhofer approximation. These assumptions were not accurate or representative of the sample which gave rise to large errors especially for material with very small particles. As computing power increased, it then became possible to apply Mie theory. This allows for more accurate results for a wider range of sample types over a larger size range. Mie theory models are based on scattering from particles and account for refraction of light as opposed Fraunhofer which is based on scattering from slits and discs and only considers diffraction from surfaces. All particle size analysis techniques measure some property of a particle and reports results as the equivalent spherical diameter based on this measured parameter. Whatever method we choose we need to be aware that different techniques will give different results. This is because we are measuring a different property of the particle (e.g. Length, Stokes’ diameter or a volume). [2] For context, in industrial applications, whilst not completely accurate, they were replacing a sieve stack, results from which were used to control massive processes such as milling.
I do recall early in the 1990s when Intel launched the Pentium chip into PCs, advice from Malvern was not to install any Mastersizers with a Pentium PC. The Mie Theory is computationally complex. Not long after it got the tick of approval, allegedly, the first Pentiums calculated the Mie Theory algorithm erroneously. I note this to highlight maybe we shouldn’t automatically accept a result.
Do you understand the principle of measurement?
As noted by Aligned Genetics, it is fundamental to understand the underlying operating principles. If you have ever used a UV/Vis spectrophotometer you will understand that there are limits in concentration, sample size, accuracy depending on the complexity of the device. There are beasts of systems with dual beams to measure the sample and the reference to ensure the highest of accuracy of sample absorbance. System specifications reported numerous features including absorbance range, stray light, spectral bandwidth, and wavelength accuracy which were revered as a proxy for quality of result promised. Beer lamberts law was elementary to the calculation of sample concentration. The law is most commonly expressed mathematically as: A = ε ⋅ l ⋅ c or Absorbance = Molar Extinction Coefficient x Path length x Concentration – this is quite rudimentary now, which leads me to ponder a device used frequently to measure concentrations of molecules such as DNA and RNA using only microlitres of sample with a variable pathlength and the system adjusts for concentration automatically. It has long been a source of puzzlement for me as I failed to grasp whether it actually was accurate. Turns out I am not the only one to question it. Apparently, it does have flaws but, how many users know of these? If you want to learn more, I found a refreshing yet comprehensive guide from Assistant Professor of Chemistry and Biochemistry, Brianna Bibel [3].
Drilling down on the nuances of the measurement principle can help to minimise sources of error.
Surely, it can’t be me!
If your sampling is poor – then you are likely a major source of error.
Sampling / sample preparation is the primary cause of variability in results across a huge range of technologies.
So many techniques have a time specific method that ensures a reproducible result. Accuracy in this case is in the eye of the beholder. If time changes the result a standard procedure is generally generated, this could be a part of the product’s control program, where it controls this variable. A firm understanding of your sample is required to be able to ensure what result you gain is valid, sometimes it can be confirmed with an orthogonal method before you ‘lock it in’.
You have 1 mg of sample to represent a 10-tonne load! Things get tricky here. Consider a truck load of wheat that has travelled from a distant wheat station to the grain handlers’ silos. When unloaded, a plume of dust thickens the air. All this is a result of settling of fines to the bottom – a phenomenon referred to as comminution. There is little that can alleviate this at the unloading step. The sample needs to be thoroughly mixed to ensure a representative sample is gained – this would be a challenge for 30 tonne load on a semi-trailer.
Depending on what is being measured, there are likely standard methods where variables are managed to gain some continuity across an industry segment – the actual result may be elusive – but that numerical value is irrelevant if direct comparisons can be repeatably made. In that sieve stack example- I recall having to design a laser diffraction report to show equivalent sieve sizes to transition the industry.
Historical standards are important as they have formed the basis of a particular measurement for a long time. Units of measure have progressively moved from a ‘natural’ measure (eg a foot, finger, or palm length) to being defined by a primary source of measurement, eg a “gold standard” stored in a secure place that all measures are referenced to. Issues arise when these standards start to change prompting the adoption of physical characteristics rather than some natural characteristic – it took mass a while to move to this, but for completeness, it is now: the kilogram is defined by fixing the exact numerical value of Planck’s constant at (h) 6.62607015 x 10-34 kg x m2 x s-1. The SI unit was largely adopted to facilitate global trade (and a whole bunch of political stuff – too much to mention here).
What about concentration and the measurement limit of the device.
If you are in the camp of “chuck in a sample – it’ll be alright” you may need to rethink this strategy. Most technologies rely on specific boundaries of sample concentration for a valid measurement; the method’s principle often defines the ‘sweet spot’. If you deviate from the defined range – expect errors. Too low and the statistical error raises, if too high, it may be catastrophic for the results – think of a technique that measures light – imagine if you block it with way too much sample – your metric of interest will likely be collateral damage. Again, understand how it works!
Garbage in: Garbage out
If analytical measurements are part of your day, I suggest you wake in the morning and recite the mantra: ‘Garbage in: Garbage out’ repeatedly.
Debris in your sample is a real issue- it can completely ruin your analysis. Many sampling protocols call for a sample pre-treatment step to ensure validity. It may be a filtration process, a centrifugation, chemical treatment, or dilution to name a few. In an optical based system, debris can grossly vary results via impeding light, contributing to size distribution, causing multiple scattering, or simply being confused by system as the target to count.
Is there a sanity check?
In many imaging systems there is a capacity to target the point of interest by gating out unwanted material identified as debris, you can threshold a fluorescence level to include / exclude a population, you may be able to control the level of declustering. You have set limits maybe as a protocol or a best guess and kicked off the analysis. You gather the result, but, have your interventions worked? Can you test the theory? Hopefully you can see the entire count population- check how the system has tagged your target- did it largely mark only what you wanted? Did it miss some? Did it include unwanted features?
Think about what the measurement is and the method, often there is a way to ensure the legitimacy of your analysis.
Errors accumulate
Pondering the opportunity for errors noted above, I firmly believe errors accumulate. If there is something wrong with a result, it could be a number of things, rarely is it one thing in isolation. Once computational algorithms are employed – such errors may be amplified.
You may need to check your understanding of a method if you are blindly adding a sample and expecting magic to happen. They all seem to output a result – if you fail to question it, I question if you have failed.