Deriving State-level Estimates from Three National Surveys: A Statistical Assessment and State Tabulations

Lisa Alecxih and John Corea

The Lewin Group

David Marker
Westat, Inc.

This report was prepared under contract #HHS-100-96-0012 between U.S. Department of Health and Human Services (HHS), Office of Disability, Aging and Long-Term Care Policy (DALTCP) and the Lewin Group. For additional information about this subject, you can visit the DALTCP home page at http://aspe.hhs.gov/_/office_specific/daltcp.cfm or contact the office at HHS/ASPE/DALTCP, Room 424E, H.H. Humphrey Building, 200 Independence Avenue, S.W., Washington, D.C. 20201. The e-mail address is: webmaster.DALTCP@hhs.gov. The Project Officer was William Marton.

The opinions and views expressed in this report are those of the authors. They do not necessarily reflect the views of the Department of Health and Human Services, the contractor or any other funding organization.

This report assesses the statistical issues involved in the production of state-level estimates related to health and welfare issues from three national surveys: the Current Population Survey (CPS), the Survey of Income and Program Participation (SIPP), and the National Health Interview Survey (NHIS). With the devolution of many welfare programs from the Federal Government to the states, there is a strong interest in being able to track the health and welfare of the population in each state. This would allow for examination of the effect of various state welfare initiatives that are to be implemented in the next few years.

STATISTICAL ISSUES

Ideally, the CPS, SIPP, and NHIS would be able to provide "direct" estimates of adequate precision for every state, as opposed to the "indirect" estimates derived from statistical models. These surveys are not large enough to produce accurate direct estimates for every state. The relevant statistical issues involved in making state-level estimates from the CPS, SIPP and NHIS include state stratification, nonsampling errors, and precision of the estimates (see the glossary for additional explanation of these and other highlighted statistical terms used in this report).

A key factor in producing direct estimates for states is the need to select the sample from strata that respect state boundaries. When strata cross state boundaries, state influences are problematic. The CPS and the redesigned 1995 NHIS strata respect state boundaries, while the SIPP does not.

The precision of a direct estimator is a function of how variable it is in the state's population and the effective sample size. The precision of an estimate for a characteristic that is highly variable in the population will be less than that for a characteristic that is not. Similarly, a larger effective sample size will provide more accurate estimates than a smaller effective sample size. Effective sample size depends on both the actual sample size and how the sample was drawn.

For all three surveys, the states are not all allocated the same sample size. Rather, the allocation of sample size to the states is made with the aim of balancing the precision requirements of both state and national estimates. As a result, there are great disparities in sample size by state. Further, these disparities are not constant across the surveys. For instance, the March 1996 CPS includes about 11 California cases for each case from the District of Columbia, while the 1993 SIPP includes 60 California cases per District case. As a result, the precision of CPS estimates for California are 3.5 times greater than for D.C., while for SIPP they are 7.5 times greater.

It is impossible to define a single level of precision that is necessary for all estimates. The level of precision that is necessary depends on the use of the estimates. Different Federal agencies have different standards for their data. Some have standards that only determine the level of precision for estimates to be used in analyses, while others have standards for precision for publication.

ABILITY OF THE THREE NATIONAL SURVEYS TO PRODUCE STATE ESTIMATES

For this report, the most current publicly available databases for two of the three surveys were examined: March 1996 for the CPS and the 1993 panel for the SIPP. In 1995, the NHIS sample was completely redesigned, so examining the 1994 data would yield little information on the ability of future years to provide state-level estimates. Thus, we only include general discussions of the ability of the NHIS to provide the desired estimates.

The two surveys were assessed based on their ability to produce four specific estimates, all expressed as a percent of the relevant population: individuals in households with income below the poverty line, individuals receiving Aid to Families with Dependent Children (AFDC)¹ individuals covered by employer-provided health insurance, and individuals with a disability. Each estimate was examined for the total population, and also for subpopulations of blacks, Hispanics, children, and the elderly.

The proportion receiving AFDC and the proportion with a work disability (except for the elderly) are both generally around 10 percent or less. These two characteristics, required the 95 percent confidence interval to be no wider than + 6.0 percentage points, following a rule of thumb that has been adopted by the National Center for Health Statistics characteristics. For the other characteristics, which are more common, we required the 95 percent confidence level to be no wide than + 10 percentage points.

Given the precision requirements used, it is possible to estimate the proportion of the total population in a state with a characteristic for almost all states from either survey. For the CPS this is also true for children and, except for Alaska, the elderly. The CPS is only able to support estimates for blacks in about half of the states, and for Hispanics in about 30 to 40 percent, depending on the measure. The smaller SIPP can support estimates for children and the elderly in the majority of states. For blacks it can support estimates in about 20 states, and for Hispanics in less than 10 states. The relatively small numbers of states for which minority estimates are supported partly reflects the geographic dispersion of minorities.

The 1996 CPS supports estimates of all of the selected characteristics for the subgroups examined at the specified precision criteria for eight states -- California, Florida, Illinois, Massachusetts, New Jersey, New York, Pennsylvania, and Texas. The SIPP supports all estimates for six states -- California, Florida, Illinois, New Jersey, New York, and Texas. The binding constraint for the data for a number of states is the sample size for Hispanics. If the selected characteristics for Hispanics are not included in assessing which states meet all of the criteria, 16 states are added for the CPS and three states for the SIPP. For the SIPP, work disability among those aged 65 to 69 also caused several states to fail to meet all of the criteria.

It is important to repeat that the precision requirements used in this report are quite arbitrary. If narrower confidence intervals are desired, the number of states meeting the cut-off will be reduced.

OTHER APPROACHES FOR PRODUCING STATE ESTIMATES

A number of alternative approaches to overcome the sample size limitations of these surveys could be pursued. These approaches include:

• Supplementary state samples -- We identified three methods of increasing state sample sizes. Each requires advanced planning and additional funding, but would permit direct estimation for states with insufficient samples in the current surveys.

  1. Increase samples -- The most straightforward procedure is to increase the sample sizes for existing surveys in the states which currently have insufficient sample sizes. Using the existing primary sampling units (PSUs) would keep costs down, but may fail to accurately represent an entire state.

  2. Dual frame approach -- A dual frame approach that combines existing in-person interview samples with telephone interviews with a supplemental sample (some or all of which could be outside the existing PSUs) provides a less costly alternative to additional in-person interviews. Limiting the supplemental sample to households with telephones requires a decision regarding whether to use an unbiased estimator that weights households separately by whether or not they have a telephone, or a biased estimator with smaller variance that disregards this factor.

  3. Add questions to the National Immunization Survey (NIS) -- If the need for state estimates can be satisfied by the addition of a few questions, it may be economical to add them to the NIS screen of 900,000 households. The telephone screen of the NIS faces similar issues regarding special weighting to retain unbiased estimators as the dual frame approach.

• Combining data from multiple years of the same survey -- A relatively inexpensive method for improving the accuracy of state estimates is to combine data from multiple years of the same survey. The precision gained is somewhat less than could be gained by doubling the sample size because the samples for each year will typically make use of the same PSUs. This may work well for characteristics that are stable over long periods, but will not work well for more volatile characteristics. Because policy makers are interested in changes in outcome measures that reflect changes in policy, combining data for multiple years may often be unsatisfactory.

• Combining data from two or more surveys -- An alternative approach is to use the data from the two or more surveys to produce a combined estimate. Unbiased estimates can be produced for each state from the CPS and NHIS. State weights are being produced for SIPP that will hopefully have minimal bias. These can be combined to produce a single estimator. While there are a number of methods for producing such a combined estimator, the most logical procedure is to weight the individual survey estimators in inverse proportion to their mean square errors. This gives greatest strength to the estimate from the survey with the most precise estimate for that state. When combining data from multiple surveys it is very important to examine nonsampling errors.

• Using indirect (model-dependent) estimators -- An advantage of indirect estimators is that sometimes when it is impossible to accurately produce estimates for individual states, it is still possible to develop useful models that describe the differences observed across a set of states. Thus, if groups of states implement similar programs it may be possible to model the effect of different types of programs, even while not being able to make accurate estimates for individual states. A limitation on the current use of indirect estimators for measuring the effect of the devolution of programs is that the only data available to develop the models are pre-devolution data. The utility of indirect estimators may increase in the future as states gain experience implementing their new programs.

IMPLICATIONS OF FINDINGS

For several reasons, it may be misleading, or even counterproductive, to require an estimate to meet a standard level of precision to be considered useful. First, using a standard may create the illusion that estimates just meeting the standard are error free, and those that fall just below the standard are entirely uninformative. Second, decision makers often have little choice but to use the best information available, even if it is poor, and an estimate that has "substandard" precision may be the best available. Third, estimates that have low precision can sometimes be usefully combined with other imprecise information to obtain more useful results

In sum, the use of the statistic must be considered in combination with the level of precision to determine the validity of an estimate. This observation lends itself to "rules of thumb" for different types of analyses, but precludes more general ones.

NOTES

1. AFDC has now been replaced by Transitional Assistance for Needy Families. This report refers to AFDC since all existing data from these three surveys reports on this program.