Would you like age-specific field reliability of your products and their service parts? Age-specific field reliability is useful for reliability prediction, diagnoses, forecasting, warranty reserves, spares stock levels, warranty extensions, and recalls. Nonparametric estimation of age-specific field reliability is easy, if you track parts or products by name and serial number for life data. What if there are life limits? What if there’s no life data?
Fortunately, Generally Accepted Accounting Principles (GAAP) require data that includes population ships (installed base by age) and returns counts by calendar accounting period. That data is statistically sufficient to estimate reliability functions, including life limits and repairable or renewal processes, without life data.
People say, “It can’t be done.”
“It’s too hard to figure out the probability that a return came from a computer made in an earlier year.” [Apple Computer geophysicist]
“…estimation requires detailed individual patient data on the time from admission (or illness onset) to death or full recovery.” [Yu et al. SARS]
“For device failures, the year of device implant was not known.” “An individual patient’s risk…could not be sufficiently evaluated in this study, such as device model and years since implant.” [Maisel ICDs]
“Accurate assessment of actual device performance is not possible based on limitations of the post-marketing surveillance system for medical devices.” [Estes AEDs]
“…as well as the failure to maintain life cycle part and labor histories at the end item level, makes it difficult to apply standard “economic useful life” models…” [RAND M88-A1 Tanks]
“The data required to provide an expected replacement profile are not automatically available at this time; thus, the expansion of this theory into full practical applications will require future efforts in developing reliable bases of information.” [James A. G. Krupp, J. Bus. Forecasting, 1993]
“One of the most abused and potentially misleading field failure study methods is the manufacturer’s warranty field return study.” [William Goble, Exida]
Nonparametric Reliability Estimation from Ships and Returns Counts
Imagine ships counts are Poisson inputs to an M(t)/G/∞ self-service queue with Poisson inputs at rate λ(t), lifetime has distribution G(t), and queue outputs are parts or product returns. It’s hard to dispute inputs have nonstationary Poisson distribution. Poisson inputs imply output counts have Poisson[λ(t-S)] distribution where S is lifetime random variable with reliability function 1-G(t), t=1,2,…life limit. [Eick et al., George 1993, Mirasol] Gary Chan recognized the same M(t)/G/∞ model for estimating (biostatistical) survival functions.
The ASQC Reliability Review 1993 article describes the nonparametric maximum likelihood reliability function estimator from ships and returns counts. An Accendo Reliability article on renewal processes shows how to make nonparametric least-squares reliability function estimates from ships and returns counts, without knowing how many prior renewals each return has experienced. [George, 2021]
Complications include: life limits (RXBandz, EpiPen, and gas turbine engine parts), batched returns (Terumo and MicroFlex gloves), sell-through time (UniPhase and Spectra-Physics lasers), return reporting and recording time delays (System General, ReliaSoft data), optimal opportunistic maintenance (engines), NTFs (Wabash crank sensors), multiple modes (failure modes, corona virus), warranty expiration anticipation phenomena, shotgun repairs, and field vs. operating time.
Complaints, repairs, replacements, spare parts’ sales may not be identified by ship date. Sell-through time may be incurred after ship date before first use. Failed gloves may be batched before being returned to manufacturer. Failures may not be reported or recorded until convenient. Life limits implicitly censor products’ or parts’ lives. NTFs mix the failures and non-failures. Corona virus cases terminate in death or recovery and both counts are recorded.
David Mortin asked, “Can it (reliability estimators from ships and returns counts) provide a reliability prediction for non-operational conditions?” Yes, depending on “non-operational” conditions: sell-through time (from production to first use), return time (from failure to return time), or operating hours per calendar time. [https://sites.google.com/site/fieldreliability/would-you-like-constant-failure-rate, 2017. Convert From Calendar to Operating Time Reliability,” ASQ Reliability Review, Vol. 18, No. 3, Sept. 1998]
What if there’s life limits? Life limits remove some installed base from risk. EpiPen epinephrine dose has a 2-year limit. Why not estimate reliability, with life limits, without life data?
Reliability Estimation with Life Limits
Some examples of life limits are: flight-safety-critical parts; auto parts with recommended replacement mileage or age; RCM recommended life limits depending on the shape of the failure rate function. (Life limits may be expressed in calendar time, operating time, cycles, or combinations.)
If you have grouped life data in the form of a Nevada table 1, the nonparametric maximum likelihood reliability estimator is the Kaplan-Meier estimator. If the product or part has calendar-time life limits, then you can still use the Kaplan-Meier reliability estimator, on data in table 2.
Table 1. “Nevada” table contains grouped lifetime returns counts R(I,j) from ships N(i) in period i.
| Ships | Returns 1 | Returns 2 | Returns 3 | Returns 4 |
| N(1) | R(1,1) | R(1,1) | R(1,3) | R(1,4) |
| N(2) | R(2,1) | R(2,2) | R(2,3) | |
| N(3) | R(3,1) | R(3,2) | ||
| Etc. | R(4,1) | |||
| Sums | R(1,1) | R(1,1)+ R(2,1) | R(1,3)+R(2,2)+ R(3,1) | Etc. |
Table 2. Suppose there is a two-period calendar-time life limit.
| Ships | Returns 1 | Returns 2 | Returns 3 | Returns 4 |
| N(1) | R(1,1) | R(1,1) | Limited | Limited |
| N(2) | R(2,1) | R(2,2) | Limited | |
| N(3) | R(3,1) | R(3,2) | ||
| Etc. | R(4,1) | |||
| Sums | R(1,1) | R(1,1)+ R(2,1) | R(1,3)+R(2,2) | Etc. |
What if your only data is the ships column and returns sums in last row, data required by GAAP?
The likelihood of ships at Poisson rate λ(i) and returns R(j) is
ΠPoisson[R(j), Σ[λ(i)*g(j−i)]; the sum is from i=1,2,…,min(j, life limit)], the product is from j = 1,2,…number of ships, and g(j−i) is the probability density of the reliability function1-G(t). Solution uses the “pool adjacent violators” method on Σreturns/Σships or direct maximization. I use spreadsheets, a VBA function, Mathematica, or an R program to solve for ĝ(t) and corresponding reliability and failure rate functions, including calendar-time life limits.
Assume the EpiPen life limit is 4 years. Sales data in table 3 came from EpiPen price-gouging trial and estimates, and the return counts came from DrugWatch.com and news: early returns estimated 120/year, DrugWatch.com says 2880 returns since 1993. Reliability estimate is based on 2015-2020.
Table 3. EpiPen ships and returns counts; reliability in years
For maximum likelihood estimation with life limits, restrict ratios of Σreturns/Σships to cohorts still within life limits. For least-squares estimation, restrict actuarial hindcasts Σg(s)N(t−s) to cohorts N(t−s) within life limits, s = 1,2…life limit.
Reliability estimators from ships and returns counts have no sample uncertainty, because ships and returns counts are population data! Fisher information and Cramer-Rao lower bound give variance of maximum likelihood estimator, and bootstrap gives confidence limits on actuarial forecast (expected demand) and prediction limits on demand random variable for fill rates.
Sample vs. Population Field Reliability?
“Real-world data and real-world evidence are playing an increasing role in health care decisions,” [FDA]. Vaccine efficacy is defined as 1−Risk(vacc)/Risk(unvacc) or
1–Cases(vacc)/N(vacc)/Cases(unvacc)/N(unvacc). Pfizer’s COVID-19 placebo efficacy is 86.25% compared with US case rate 5.40%, because Pfizer’s untreated “convenience” sample case rate was 0.75%. Why not treat a representative sample and compare untreated population reliability?
How could you compare the Kaplan-Meier reliability estimate (treated sample table 4) vs. the nonparametric estimate from an untreated population, without life data?Likelihood ratio, Kolmgorov-Smirnov (K-S), Kullback-Leibler divergence (K-L),… tests: require uncensored lifetime data! I fixed the K-S test to deal with censored lifetime data, [George 1997], but that fix didn’t account for life-limits, so I simulated samples with the same period sums (table 5) for comparison by the Kolmgorov-Smirnov test.
Table 4. Grouped life test data
| Period | Cases | Deaths Period 1 | Deaths Period 2 |
| 1 | 98 | 2 | 3 |
| 2 | 100 | 2 | |
| Period Sums | 198 | 2 | 5 |
Table 5. “Neutrosophic” K-S test simulated samples, [Smarandache]
| Period | 1 | 2 | 1 | 2 | 1 | 2 | 1 | 2 | 1 | 2 |
| 1 | 2 | 2 | 2 | 1 | 2 | 0 | 2 | 4 | 2 | 5 |
| 2 | 3 | 4 | 5 | 1 | 0 | |||||
| Sums | 2 | 5 | 2 | 5 | 2 | 5 | 2 | 5 | 2 | 5 |
Table 5 shows all simulated population life data with the same column sums or event counts as in table 4. Compute the K-M estimator from the simulated population life data and its K-S distance from the sample K-M survival function estimate [George 1997]. If the sample K-M estimator K-S distance is less than some percentile of the simulated |population−sample| K-S distance, do not reject the null hypothesis.
Free offer
If you would like to estimate field reliability or test hypotheses with life limits, but without life data, send data to pstlarry@yahoo.com and describe it. I will send back workbooks with estimates and uncertainty.
References
Chan, Kuen Chuen Gary, “Survival analysis without survival data: connecting length-biased and case-control data,” Biometrika. 2013; 100(3): 10.1093/biomet/ast008., doi: 10.1093/biomet/ast008
S. G. Eick, W. A. Massey and W. Whitt, “The Physics of the M(t)/G/∞ Queue,” Ops. Res., 41, 731-742, 1993
Mark Estes III, N. A. MD, “Automated external defibrillators-device reliability and clinical benefits,” JAMA, Vol. 296, No. 6, pp. 700-702, August 2006
FDA, “Submitting Documents Using Real-World Data and Real-World Evidence to FDA for Drugs and Biologics Guidance for Industry,” May 2019
George, L. L. “Estimate Reliability Functions Without Life Data,” Reliability Review, ASQC, Vol. 13, No. 1, March 1993
George, L. L. “Product Reliability Comparison with Censored Data,” or “To the Man With a Hammer, Everything Looks Like a Nail,” ASQ Reliability Review, Vol. 17, No. 1, March 1997
George, L. L., “Renewal Process Estimation, Without Life Data,” www.accendoreliability.com Weekly Report, Sept. 2021
Goble, William M., “Field Failure Data – the Good, the Bad, and the Ugly,” Exida, 2005-2012 Version 2.4, Feb. 2012
Kaplan, E. L. and P. Meier, “Nonparametric estimation from incomplete observations,” J. Amer. Statist. Assn., Vol. 53, pp 457-481, June 1958
Krupp, J. A. G., “Forecasting for the Automotive Aftermarket,” The J. of Bus. Forecasting, pp. 8−12, Winter, 1993−94
Maisel, William H MD, MPH, “Pacemaker and ICD generator reliability. Meta-analysis of device registries,” JAMA, 2006;295:1929-1934
Mirasol, N. M., The Output of an M/G/∞ Queuing System is Poisson,”Operations Research,”11, 282-284, 1963
RAND Corp., “Improving the Army’s Management of Reparable Spare Parts,” by John R. Folkeson and Marygail K. Brauner, RAND MG-205, 2005
Smarandache, Florentin, Introduction to Neutrosophic Statistics, Sitech & Education Publishing, Columbus, Ohio, 2014
Yu, Philip L. H., Jennifer S. K. Chan, and Wing K. Fung, “Statistical Exploration from SARS,” The Am. Statistician, Vol. 60, No. 1, pp. 81-91, Feb. 2006
Thanks to Fred for publishing the article and for rendering all the Greek letters and symbols. I appreciate the opportunity to report on real reliability. I am grateful to the people who give me such interesting problems.
Thanks to Fred for stumping me for a while. The lede quotation of this Weekly Update included “I am well are,…” That stumped me for a while. It should have said “I am well aware,…”