Validate Jackknife Replication

Introduction - We developed a Vitalnet module - VB-QMS - for analyzing BRFSS data. VB-QMS makes BRFSS data analysis much better, easier, faster, and more reliable. Due to the complex survey design, confidence intervals (CI) for BRFSS data are non-trivial. Several CI methods are available for BRFSS data. SAS and SUDAAN typically use Taylor Series linearization (TS). However, we chose "Jackknife Replication" (JR) because 1) JR can calculate a CI for any outcome, including medians, and 2) JR is easier to understand and explain.

• Jackknife Replication (JR) - Jackknife Replication recalculates the outcome, such as "% Yes", many times, each time leaving out one or a few observations, and re-weighting the remaining observations. Each recalculation is called a "replicate". Then, JR calculates the confidence interval based on the distribution of the replicate outcomes. The method is called "jackknife" because it is so generally useful, like a jackknife.

• Taylor Series Linearization (TS) - The other main option, "Taylor series linearization", converts each observation to a "linearized variable". Then, it calculates the confidence interval based on the distribution of the linearized variables. Unfortunately, TS cannot calculate confidence intervals for medians. Also, the details of the TS algorithm are poorly documented. So currently, TR is essentially a "black box".

• Vitalnet (VB-QMS) implementation - JR is computation-intensive and can be slow. However, we have optimized JR to make it much faster (typically a few seconds). Also, we have determined that a JR CI with smaller numbers of replicates is close to one with an unlimited number of replicates. In essence, for exploratory data analysis, the user can select a smaller number of replicates. For published results, maximum replicates are recommended. If possible to obtain details of the TS algorithm, we may add TS as an option in the future, to speed up CI analyses in some cases with very large numbers of records.

• Why Validation was Done - We take data quality and correct results very seriously. To ensure the JR programming was done correctly, and to verify that JR and TS give very similar results, we systematically compared a series of VB-QMS CI calculations with ones calculated by CDC WEAT, and several prominent State systems.

---

Methods - For each comparison, a 95% CI was calculated for "% Yes" (or % No), for a particular BRFSS variable analyzed by VB-QMS, WEAT, and one of two State systems, for 2005, for both male and female. A range of percents and response counts were used, to expose any difference for larger or smaller percents or response counts. One digit after the decimal point was used, because this is the maximum precision of WEAT and some State systems. Maximum replicates was used for the JR method. The BRFSS variables were picked at random, and to ensure that all three systems (VB-QMS, WEAT, and the State system) included the variable. Once a comparison was done, it was included, regardless of the results. We used two State systems that we knew existed and functioned: TX System and UT System.

---

Results - Each Table A-F below shows a comparison between WEAT, VB-QMS, and a State system. When only one system had a different CI, or when a system had a markedly different result, the differing number is marked in red.

• Same Percents and Counts - All the systems gave the same basic results. For example, 502 Yes and 23.2 % Yes for male binge drinking question.

• Very Similar VB-QMS and WEAT CI Results - We found very close results between VB-QMS and WEAT. Other than the extreme case in Table F with very low (49) male counts, they differed by only 0.1 and in just a few cases. Also, VB-QMS (and WEAT) were the "odd man out" (colored red) only 2 of 24 possible times, again showing very similar.

• State Systems CI Results also Similar - The State systems were "odd man out" (colored red) much more often, 16 of 24 possible times. However, only with low numbers (Tables E and F) were differences larger than 0.1 seen. So in practice, the differences would probably not significantly change the interpretation of results.

---

Discussion - The JR method produces valid confidence intervals for BRFSS data, and is correctly implemented by VB-QMS.

• Why any Differences? - VB-QMS uses jackknife replication (JR) to calculate the CI, and the procedure is fully documented. It is not surprising that VB-QMS differs a little from WEAT, since WEAT uses Taylor series linearization (TS), a totally different method. If anything, the surprise is that WEAT and VB-QMS are very close.

• Mostly Differ with Small N - Only in Table F with only 49 responses do the VB-QMS and WEAT results differ more than 0.1 This is not surprising: the low number of events expose some difference between the algorithms. One could guess that JR is "right", since it is an exact non-parametric method. In any case, the differences are still small, 0.3 at most, and less than the State systems differed.

---

Table A) _RFBING3 (Binge Drinker), TX

Table B) _BMI4CAT (Overweight), TX

Table C) _RFSMOK3 (Current Smoker), TX

Table D) _FV5SRV (Five a Day), TX

Table E) _CHOLCHK (Chol. past 5 years), UT, 45-49

Table F) _CHOLCHK (Chol. past 5 years), UT, 18-24

References consulted concerning TS and JR methods include the following: • Analysis of Health Surveys, Korn and Graubard (1999) • Analysis of Survey Data, Chambers and Skinner (2003) • Analyzing Complex Survey Data, Lee and Forthofer (2005) • Introduction to Variance Estimation, Wolter (2007) • Pitfalls of Using Standard Statistical Packages for Sample Survey Data, Brogan (1998) • Practical Methods for Design and Analysis of Complex Surveys, Lehtonen and Pahkinen (1995) • Sampling Error Estimation for Survey Data, Brogan (2005)

Abbreviations used in the tables and text are:

• CI = confidence interval
• LL = lower confidence limit
• UL = upper confidence limit
• TS = Taylor series linearization
• JR = jackknife replication

Contact us if you have suggestions / comments, or to license Vitalnet BRFSS Query & Mapping Software to make better use of your data.