Conducting Community Surveys A Practical Guide for Law Enforcement Agencies Conducting Community Surveys A Practical Guide for Law enforcement Agencies by Deborah Weisel October 1999, NCJ 178246 A joint project by the Bureau of Justice Statistics and the Office of Community Oriented Policing Services U.S. Department of Justice State and Federal Corrections Information Systems Acknowledgments This report was written by Deborah Weisel. It was prepared under the supervision of Marshall DeBerry and Steven K. Smith, Ph.D., of the Bureau of Justice Statistics and by Pam Cammarata of the Office of Community Oriented Policing Services. Meg Townsend, formerly with the COPS office, also contributed to the project. The project was supported by BJS under contract number OJP-99-135-M. Tom Hester, Lea Gifford, and Marika Litras of BJS provided editorial review. Jayne Robinson of BJS administered final publication. The Crime Victimization Software may be obtained by contacting 1-800-732-3277 or ASKCVS@ncjrs.org. Contents Why undertake a community survey How was the survey developed? What is the local-level Crime Victimization Survey? How have surveys been used by police Use of telephone surveys Overcoming fears of surveys Survey development Thinking about and setting goals What do you really want to know from the survey Tailoring the survey to fit your needs Modifying the level of analysis Modifications to reflect local conditions Modifications to incorporate measures of planned police initiatives Modifications to survey should be consistent and neutral Who ya gonna call? The critical issue of sample size Population and samples Issues of probability sampling Generating the sample frame What is the appropriate sample size? How representative is the sample? Survey administration Carrying out a survey Coordination and quality control Selection, training, and supervision Analyzing and interpreting survey results Frequencies and crosstabulations More sophisticated analyses Conclusion Appendixes Appendix A. Terminology Appendix B. References Appendix C. Seven basic steps for conducting telephone surveys Appendix D. Costs Why undertake a community survey? The widespread adoption of community policing across the nation has increased the interest in law enforcement agencies' conducting community surveys. Many police engaged in community policing want to know "how we're doing" from the citizens' perspective. Community surveys provide descriptive information that goes beyond the traditional measures of police workload, arrest activity, reported offenses, and calls for service. Besides, since community police officers are trying to be responsive to community concerns, any complete measure of success would have to include asking the members of the community themselves. Community surveys at a city-wide or county-wide level can be designed to provide police with reliable feedback from citizens about perceptions of police performance. In addition, these surveys collect information about criminal victimization, residents' views about crime, and their willingness to report crime to the police. From a police management perspective, these surveys collect information about the most effective approaches to dealing with crime. Since these surveys provide a measure of police performance, they can be used to analyze the way police deliver services and possibly change the allocation of resources where needed. Some community surveys provide detailed information about specific problems affecting parts of a city, which helps in focusing police resources. Survey research is a science and requires that accepted practices be followed. But carrying out surveys need not be intimidating. Even terms such as field testing, validation, sampling and statistical analyses need not be barriers to you if you want to conduct survey research to shed light on police practices. You can partner with a local college, university, or research firm to assist you in developing and administering the survey as well as reporting the findings. Local colleges, universities, and research firms can assist you with survey issues addressed throughout this guide, including validity, reliability, purpose, probability sampling, survey administration, and analyzing and interpreting survey results. Survey research is straightforward, although it can be time-consuming to follow acceptable practices. To make available good community surveys and reliable survey methods to cities, counties, and states, the Office of Community Oriented Policing Services (COPS) and the Bureau of Justice Statistics (BJS) in the U.S. Department of Justice have developed a software package that includes a standardized community survey you can administer by telephone. The entire questionnaire is included at no cost in the software package available to government agencies and researchers. This guide comes with the software and its technical how-to manual. It gives practical, basic pointers for police in conducting community surveys. It provides an overview of key issues involved in conducting survey research. This guide provides some basic do's and don'ts for conducting surveys that can withstand close scrutiny by police leaders, academicians, community groups, politicians, and the media and that can be compared with similar surveys by other jurisdictions. The goal of this guide is to help you identify the issues in conducting useful surveys. However, this guide addresses only the basic elements of survey research. To learn more about research and statistics for police managers, see Appendix B. Police agencies may also consult with professionals about the technical aspects of survey research. The step-by-step procedures in this guide will help you and whoever will conduct the survey to address the following basic but critical issues: Why are we doing a survey and what do we want to know? What kind of resources and commitment do we have for this effort? Who should be surveyed and how? How many people should be surveyed? What do we do with the responses to the survey? Answering these questions will make conducting surveys a relatively painless process to collect valuable information from the community. This guide also provides the reader with a brief description of the critical issues affecting community surveys. How was the survey developed? Some background on this survey project is in order. The community survey discussed in this guide builds on the work currently part of the National Crime Victimization Survey, or NCVS. The NCVS has been conducted since 1972 by the Bureau of the Census for the Bureau of Justice Statistics. The results of the NCVS are widely available to police and others, and are developed to provide national estimates of victimization, illuminating how actual victimization varies from crime reported to the police. Because much crime is not reported to police, the NCVS findings clarify patterns of crime that are actually experienced by citizens and give the victim's perspective on what happened and the consequences. Because the NCVS is based on a nationwide sample designed to produce estimates for the entire United States, the NCVS does not provide insight into crime problems at the city, county, or state level. To assist localities with collecting this important victimization information and other information at the city level, COPS and BJS have developed a standardized software package. Known as the Crime Victimization Survey or CVS, the CVS software duplicates all the questions in the NCVS. In addition, the CVS includes a component which asks specific questions about community policing. Moreover, both the victimization component and the community policing component of the CVS can be modified to meet specific local needs. What is the local-level Crime Victimization Survey? The CVS is a generic telephone survey questionnaire covering what police engaged in community policing often seek to learn from the community: What is the extent of residents' exposure to crime and perceptions of crime? What are local perceptions of community disorder and quality of life? What are local priorities for addressing neighborhood conditions of disorder? How fearful are residents of crime? What self-protective steps have been taken by citizens? What is the extent and nature of police contact with residents? What are citizens' perceptions of police activities? How satisfied are citizens with police performance? What are public attitudes toward and knowledge of community policing? The community policing questions supplement the core victimization questions taken from the NCVS. The core victimization questions from NCVS examine in great detail the nature of victimization experiences within a household -- identifying victim characteristics, victim behaviors, offense characteristics, reporting behavior including an explanation of why the incident may have been unreported, offender characteristics, use of weapons, and a host of other information about the victimization experience. The reader should be cautioned that the CVS is a basic or generic community survey instrument. Survey research is not typically "one size fits all." Later in this guide, we'll discuss how to revise the basic survey to fit the specific needs of your jurisdiction. There are important benefits to conducting a standardized survey, however. The principal benefit is that many of the questions related to scientific methodology have been addressed by experts. You can have greater confidence that the results of the survey are valid. A second reason is that it is possible to make comparisons with other jurisdictions which have conducted similar surveys. For many situations such as addressing crime and disorder problems, such comparisons may not be helpful. However, to interpret the measures of citizen satisfaction with your department resulting from this survey, it may be useful to compare your agency to others. How have surveys been used by police? Police have long been interested in citizens' views of police. To evaluate police programs, academic researchers have used surveys of citizens -- in person, by telephone or by mail, with different response rates and costs. But police agencies have only intermittently surveyed citizens, with some increase in these surveys beginning in the 1980s. During the 1980s police began to conduct their own -- albeit often informal -- surveys. Many of these surveys were door-to-door surveys designed to document the extent and nature of fear of crime. Among others, fear reduction programs by police in Houston, Newark and Baltimore County made use of community surveys. During the late 1980s, many police came to view the public as customers or consumers of police service. Police often used citizen or customer satisfaction surveys to gauge the reaction of citizens to contact with police. These surveys were often mailed surveys, sent to citizens who had complained, received traffic citations, or had other formal contact with police. Telephone surveys were also used for this purpose. As part of the trend towards community policing and problem-solving in the mid- and late 1990s, community surveys have increasingly been used to collect local-level information from citizens about specific problems in their communities. For the most part, these surveys have been administered to sub-units within cities -- public housing developments, neighborhoods or other geographic entities. Many of these surveys have been administered face-to-face at the homes of residents; however, door-to-door surveys are very costly to carry out. While all of these approaches to collecting information from citizens have been used, surveys have not always provided reliable information to police. Because of the way people were selected for surveys, police surveys have typically not met scientific standards of reliability, and some have been subject to criticism. To provide feedback about community policing, there has been increasing interest among police, city councils, and mayors' offices in obtaining reliable survey information that meets scientific standards by using probability samples. In fact, some cities now routinely conduct citizen surveys to provide feedback on police performance. * Scottsdale, Arizona, conducts (through a contractor) an annual citizen satisfaction survey to assess service delivery. (See the questionnaire at www://ci.scottsdale.az.us). The telephone survey queries approximately 400 citizens through a random digit dialing approach. The sampling procedure is so standardized and the survey so routinized that information is used to guide the city's budgetary process. * Similarly, police in Reno, Nevada, completed their 17th semi-annual community survey in early 1999. The Reno department primarily uses volunteers, such as university students and seniors, to carry out the survey of 430 citizens. (For more information contact the deputy chief for planning, training and research via http://www.reno.gov) Use of telephone surveys As a method of research, telephone surveys have many benefits. The telephone interview is typically non-threatening to the respondent and allows the respondent to feel the degree of confidentiality or anonymity assured by the interviewer. Telephone surveys also preclude the need for interviewers to conduct interviews in high crime areas where the risks to personal safety may be high. Typically, telephone surveys are inexpensive relative to other types of survey research, and response rates to telephone surveys are generally higher. Despite the strengths of telephone surveys, there are some limitations which should not be overlooked. Although the vast majority of Americans have telephones in their homes, some do not, and in some neighborhoods the proportion of households with phones is low. Telephone surveys are inherently biased against lower-income citizens, since a larger proportion of low-income families have no telephone. Telephone surveys may also underrepresent persons with language barriers, in some ethnic groups or in age groups like the very young or the very old. Some groups such as the homeless, young inner-city males, or college students may be severely underrepresented in phone surveys. Since these persons may have different views of police service or different exposure to crime, this inherent bias in telephone survey research should be recognized throughout the survey process. There are procedures to weight demographic groups which are not fully represented in telephone surveys. These procedures are discussed later in this guide. Overcoming fears of surveys Conducting surveys can be intimidating and for some good reasons. The results of community surveys come under a great deal of scrutiny by media, politicians, community groups, academicians, and police leaders. If police or sheriff's departments develop their own surveys, they have to be concerned about their perceived bias as survey administrators. Surveys can be viewed as self-serving devices, designed to make the police look good, especially if there are concerns about police brutality, police attention to minority areas or police treatment of minorities, and other sensitive issues. To address perceived bias, police must focus attention on every aspect of survey research, including question wording, field testing of questionnaires to identify problems, and other processes of survey development. In addition, police must identify an unbiased sampling frame and develop procedures for contacting prospective respondents. This is a big task -- one with which a local university or college can assist. Using the CVS and a standardized sampling procedure will relieve many of the external and internal pressures on police in conducting surveys. By following sound sampling procedures, city-level findings can be comparable to those of other jurisdictions. Replicating the survey on a periodic basis will provide police agencies with regular and reliable information about citizens' views of crime and public safety. The CVS eliminates the need for many of the procedures that occupy survey researchers. The survey instrument has already been rigorously pre-tested to ensure that questions are neutral and understandable, and responses have been scaled and validated. In addition, the software includes a method for generating a simple random sample of telephone numbers through a random digit dial approach by using the telephone prefixes for the area. Telephone surveys are a promising method for police to collect quality information from citizens on a routine basis and at relatively low cost. Conducting survey research that meets standards of reliability is not difficult but requires a significant amount of effort. Importantly, the commitment and resources for this task should be carefully evaluated by police before undertaking survey research. Only if sufficient resources are available to police to conduct reliable scientific surveys can the benefits of telephone surveys be fully realized. Survey development Carrying out the community survey involves installation and use of the CVS software provided by COPS/BJS. While this task should be carried out by someone knowledgeable about computers, it is a straightforward procedure that involves following clearly specified steps. The technical manual accompanying the software includes detailed installation and use instructions. Thinking about and setting goals The first and most important step of survey research, and the most often overlooked step, is determining the goals of the survey. Surveys for police can be used for different purposes and it is important to thoroughly discuss and articulate these goals prior to conducting the research. Avoiding or rushing this step will invariably result in problems. Articulating survey goals invariably saves time, money and headaches. For example, if your jurisdiction is not interested in collecting information about victimization for different types of crime, much effort would be expended for collecting data with little value to your agency. Similarly, if your agency is not engaged in community policing -- and has no interest in implementing community policing -- you won't want to waste time asking respondents about community policing initiatives. If your agency is concerned about its relationship among different groups of Asian heritage -- perhaps Lao, Cambodian and Vietnamese -- the survey must ask about country of origin. Thinking about these needs after the survey is conducted is too late. So, thinking through and articulating goals help police develop a survey instrument that is most appropriate for the jurisdiction. Articulating goals also facilitates analysis. For example, if you're interested in the rate of crime in the Northeast area of your city, the sampling plan must collect enough responses from that neighborhood to yield valid information. If you're particularly interested in the opinions of a small ethnic group, or a particular age group, this affects your sample size and sampling plan. It's best to know these needs at the beginning of the survey, not after the survey is completed. During the discussion of goals, police should also plan whether the survey is to be repeated periodically. While there are many benefits to conducting periodic surveys of citizens, this replication may be beyond the resources of the jurisdiction. It may be more feasible to conduct the community survey every other year or even every three years. While the longer time frame may wash out some of the information about police services, such a survey will still provide valuable information which is a snapshot of a point in time. If possible, decisions about replication should be made before conducting the first survey as plans to replicate the survey can have implications for what questions are asked. While your jurisdiction may not be exceptionally interested in victimization rates for a one-time survey, declines in victimization rates -- as reflected in a periodic survey -- may provide useful information for evaluating crime control and public safety initiatives. An example of this utility is often heard in discussions of community policing. Many police leaders believe that reported crime will first rise when community policing initiatives are implemented, as residents become more willing to report offenses. Victimization surveys may provide evidence for this hypothesis, revealing that as actual victimization declines reported crime rises. Without victimization information, police will be hard pressed to document this explanation of community policing impact and will be able only to provide anecdotal support for the hypothesis that crime is actually declining. So, thinking through the survey goals is a necessity for conducting good survey research. This step ensures that the survey accomplishes the following: Makes most efficient use of resources by selection of appropriate sample size Provides the most meaningful information for local needs Assists with development of an analysis plan Plans for future information needs. What do you really want to know from the survey? In practical terms, the answer to this question may be affected by knowing what kinds of things you can learn from a community survey. Among the possible objectives of a police department using the standardized CVS including the community policing section are the following: * Victimization -- differences between reported offenses and actual victimization by individual characteristics such as race, gender, age and income * Comparison of local victimization rates with those of other jurisdictions * Measures of citizen willingness to report crime * Indicators of citizen fearfulness * Analysis of crime and disorder problems * Measures of police performance: survey measures citizen knowledge and attitudes, satisfaction reflects how well the police are doing their job from the citizens' perspective *Measures of public information efforts * Comparison of a jurisdiction's community policing efforts -- and impact on victimization and fear -- with those of other jurisdictions Tailoring the survey to fit your needs The goals of the survey will affect the kinds of questions which the police agency will want included in the survey. The basic survey questionnaire -- either the CVS portion or the community policing section -- can be modified to fit different police objectives. Some questions may be deleted or altered, while additional questions may be added. The technical guide that accompanies the CVS software includes directions for carrying out these modifications. Modifying the level of analysis One modification which may be desired by local jurisdictions is to alter the level of analysis. The CVS survey is designed to collect information about a city. The survey currently does not contain area-level information smaller than a city. While city-level information is certainly easiest to collect, there is no reason why the level of analysis cannot be changed. However, if police wish to collect and analyze information about smaller areas within the city, such as neighborhoods or other geographic areas, typically a larger sample must be surveyed. Sample size is discussed inlater in this guide. Since many police agencies are interested in differences in crime and victimization in specific areas within their city, agencies may wish to add a question identifying the respondent's area within the city. Addresses are requested in the survey or respondents may also be asked to identify their zip code, neighborhood by name, region or quadrant of the city (linking the survey to a GIS may facilitate data collection by neighborhood). Police agencies may want to collect information only from specific neighborhoods rather than undertaking a city-wide survey. If information about geographic areas for telephone prefixes is available, a sampling frame may be constructed making use of this knowledge. Sample size will vary accordingly. Alternatively, interviewers may screen calls to identify only respondents who live in certain portions of the city. This task would be time consuming. Modifications to reflect local conditions Other modifications to the survey will reflect differences in jurisdictions across the nation. On occasion, these differences may consist of wording or particular localized conditions. For example, the survey asks respondents about steps they have taken to prevent crime in the home. In warm areas of the country where windows and doors may be left open, simply keeping doors and windows shut may be a deterrent to crime. While the survey asks if the respondent has taken other precautions, the current survey questions and responses should be carefully evaluated to ensure they reflect local conditions and objectives. The local crime and public safety scenario in cities varies based on a variety of conditions. For example, crime varies by housing stock since high rises feature different problems than single-family residences; crime varies by the presence of alleys or other street configurations and conditions; by topography, such as canyons; by development, such as proximity to retail areas, and by many other features. Modifications to incorporate measures of planned police initiatives An important element for modifying the questionnaire is to incorporate measures of planned police initiatives. Say, for example, the police plan to mount an extensive public information effort to encourage residents to keep windows and doors shut. If this type of effort is planned, a baseline question can be included in the survey to determine what proportion of residents currently keep windows and doors shut. This baseline measure will then provide a before-and-after measure of the impact of the police public safety measure. This research design, known as a pre-test post-test, provides excellent information about the impact of prevention or intervention efforts. Crackdowns on gangs, street-level drug dealing, truancy, public drinking and a host of other community-policing initiatives can be well documented by using this design. Modifications to survey should be consistent and neutral Modifying the questionnaire requires care and attention to question construction and response options. In general, questions should be neutral and consistent with other questions in the survey questionnaire. Response options should also be consistent with the responses in the questionnaire. For example, if concern about gangs in your jurisdiction is a major issue, the survey may be modified to ask the respondent: "How fearful are you about gangs in your city (and/or your neighborhood)?" Since response options to similar questions in the survey are worded as (1) Very fearful (2) Somewhat fearful (3) Not very fearful (4) Not at all fearful this same sequence of responses should be used for the new question. Who ya gonna call? The critical issue of sample size Once the survey has been modified to be consistent with the goals and objectives in a particular jurisdiction, attention must turn to the selection of respondents. By far, the most common question of police conducting community surveys is "How many people must I survey?" The answer to this question relates to the goals and objectives of your survey. In this section, we will provide some general guidelines for decision making. We mentioned previously that police surveys are often criticized because they are not scientific or probability samples, thus their findings are not representative of anything in particular. Investing the time to develop a defensible sampling strategy and a probability sample is an important procedure. So, a brief and very basic discussion about sampling procedures is necessary. Population and samples When you sample people, you select a portion or subset of everyone you are interested in, then make generalizations from the subset to all people of this type. The larger group is called a population while a sample is the subset or a portion of the population. If you were interested in the experiences of women who have been victims of theft, the population is all women. Since it is not possible to interview all women, you will want to interview a subset of women. The subset is your sample, and it is made up of cases. In the example so far, the cases are those women selected for interviewing. If possible, you will want to know something about the population. For example, what proportion of the population have been victims of theft in the prior six months? To answer this question with a sample, you need to be sure that the sample is representative of the population. Characteristics of the population are called parameters. In this example, the proportion of women in the population who have been sexually assaulted in the prior six months is a parameter. Though this number exists, it is unknown. You will need to estimate it from the sample. The estimate of the population parameter from the sample is called a statistic. If the sample is unrepresentative of the whole population, then the statistic you calculate is unlikely to be an accurate estimate of the population parameter you care about. For example, if you interview women at a women's crisis clinic, you may overestimate the proportion of women who have been victims of theft because it is quite likely that this sample or subgroup has a higher than average theft rate than women in the population. How do you get a representative sample from your population? All sampling begins with the translation of a defined population into a sampling frame. A sampling frame is either a list of possible cases or a procedure for finding eligible cases. It is from this sampling frame that you will select your cases. Ideally, everyone in the population should show up in the sampling frame. Say you were interested in the experience of college undergraduate women who have been victims of theft. There is one university in your jurisdiction, but there are too many female undergraduates to interview. The university might be able to provide you with a list of undergraduates for you to use as a sampling frame. Issues of probability sampling Probability sampling includes a variety of techniques that produce cost-effective data sets that are representative of the population from which they are drawn and yield information that quantitatively describes our confidence in how well the sample estimates describe population characteristics. Non-probability samples can be cheap or expensive, but we often have no basis for claiming they are representative of the population, and we are not able to measure how well our sample results reflect the whole population. A probability sample is one that is selected in such a way that each member of the population's sampling frame has a known probability of being selected and that probability is greater than zero. Note that this does not necessarily mean an equal probability of being selected, though some probability samples have that characteristic. In some circumstances it is desirable to select some members of population with certainty. If you are using something other than equal probability of selection, weights should be used to take into account the differing sampling probabilities. A non-probability sample is a sample in which you either do not know the chances that each member of the population has of being selected, or there are some members of the population that will not be selected. Interviewing the women in your office about their theft victimization experiences might help you think about the problem of women as victims of theft, but this group is unlikely to be representative of the larger population in your city. When using a probability sample, two things heavily influence the confidence in your estimates. The first is sample size. This is the number of cases in your sample. The bigger your sample, the greater the confidence that your statistic is an accurate measure of the population parameter. You have some control over your sample size, at least within the bounds of your budget or the budget of the agency funding your research. The second factor which influences confidence in estimates is the prevalence of the characteristics of interest in the population. The greater the prevalence, the smaller the sample you will need to get the same level of confidence. The more rare the characteristic is in the population, the larger the sample required to get the same level of confidence. While you have no control over the prevalence in your jurisdiction, your knowledge about how common the characteristic is in your population can be used to design your sampling strategy. Simple random sampling is one of four common probability sampling designs. Each has its advantages and disadvantages. Simple random sampling is the type of sampling design we usually think about, and the sampling strategy which is recommended in software package included with this guide. Once the sample size has been determined and the sampling frame has been developed by inputting the jurisdiction's telephone prefixes to generate a list of telephone numbers, each item on the list is given a unique number. Using the software's capacity to produce random numbers, cases (telephone numbers) are selected for the sample. Each case (telephone number) in the sampling frame (list of telephone numbers) has exactly the same probability of selection and that probability is equal to the sample size divided by the population size (number of entries in the sampling frame). It is important to include all prefixes that significantly cover the area of interest. Leaving out some will lessen the representativeness of the sample. Simple random sampling is most useful when you are not investigating rare events or small population segments. For example, if you think that only about 2 percent of the female undergraduates at a certain college have been the victim of a theft in the prior six months, then simple random sampling is a very costly approach. To be sure that we have 30 women who have been victims of theft in our sample, we would have to interview 1,500 female students. In contrast, if 40 percent of female undergraduates are likely to have been theft victims, then simple random sampling may be quite efficient. This, of course, is the difficulty for a city-wide sample for victimization; despite prominence in the media, violent crime events are relatively rare in a jurisdiction. Very few persons are victimized by violent crime. Generating the sampling frame Now that we've discussed the issues underlying determining sample size, we're back to the question -- How many are enough? Since violent crime rates -- that is, rates per 1,000 population C are fairly well established and consistent across different places, we know that violent crime is a relatively rare event. Using the CVS software, the sampling frame for the CVS and community policing survey consists of randomly generated telephone numbers which, by and large, represent separate households. (Some may represent multiple phones within a single household while others serve multiple households, such as a rooming house.) This method of telephone number generation is known as random-digit dialing or RDD. In developing a sampling frame, RDD avoids the limitations of using phone books, since many telephone numbers are unlisted. In generating the sampling frame, enough numbers must be generated to account for refusals -- that is, the prospective respondents who decline to be interviewed -- and ineligibles such as non-residential numbers, non-working numbers, facsimile, computer lines, and cell phones and numbers which duplicate other lines within a household. These ineligible numbers may constitute a very large portion of the sampling frame, particularly in moderate to high-income neighborhoods. There is no reliable information about ineligibles generated through RDD. An estimate of about 60 percent may be appropriate for purposes of planning. Actual experience will probably vary a great deal by location and depend on the effort applied to obtaining participation from the survey respondents. What is the appropriate sample size? Deciding on an appropriate sample size requires effort and planning. There are no standardized sample sizes for surveys. Recommended sample sizes are even independent of city size. Since violent crime is a relatively rare event and is somewhat equally as rare in large cities as in small cities, recommended sample sizes will not vary much based upon the total population of the city. In general, a final sample size of at least 200-250 will be the absolute minimum necessary for making inferences to the larger population. In the 12 large cities in which the CVS questions were tested, final samples consisted of approximately 800 households (in each city) to produce a sample sufficient for making inferences about victimization experience. These numbers were necessary to generate a sufficient number of victims of violent crime in the sample since violent crime victimization is statistically a rare event. In general, an appropriate sample size will fall somewhere between these points, depending of course on the objectives of the survey. If the police agency is not interested in violent crime, and desires only to collect information about property crime victims, a smaller sample may be used since property crime is statistically more common than violent crime. One way to think of selecting sample size is to develop a "best guess" of the level of victimization experience being sought in the sample. Using Table 1, say that you estimate that half of 1 percent of persons in your community experience the type of victimization you wish to examine. If your population is 500,000, this best guess of victimization suggests that about 2,500 persons are victims. To find 30 of the 2,500 victimized persons, you will need to call or interview at least 6,000 persons -- your sample size is 6,000. A minimum number of 30 cases is an absolute requirement for large sample analysis with statistical procedures normally used. ------------------------------------------------ Table 1. Best guess for sample size Best gues percent of Minimum sample size to find n people victimized cases with victimization n=30 n=50 n=100 n=150 0.5% 6,000 10,000 100,000 300,000 1% 3,000 5,000 10,000 15,000 5% 600 1,000 2,000 3,000 10% 300 500 1,000 1,500 15% 200 333 666 1,000 20% 150 250 500 750 25% 120 200 400 600 ------------------------------------------------ --------------------------------------------- Reno, Nevada Reno police conduct community surveys twice each year, and a sample size of 430 provides reliable information. In general, the larger the sample size the greater the accuracy in the survey estimates, assuming the sample is increased in a random manner. --------------------------------------------- If you are interested in looking at subgroups such as race or gender by type of victimization, then doubling the sample size is appropriate. This requires that your minimum number of cases increases proportionally as indicated in the top row of Table 1. If the subgroup of interest is a very small proportion of the overall population, then tripling or quadrupling the sample size may be in order. In these situations, simple random sampling is prohibitively expensive since you need more and more interviews C and a more efficient method of probability sampling should be used. Professional assistance should be sought under these conditions. If the victimization experience you seek to examine is more common than the previous example, you may call fewer people. For example, if the victimization experience is estimated at 15%, looking at Table 1, you will see that you only need to call 200 persons to find 30 persons with that type of experience. How representative is the sample? As telephone numbers are randomly generated for the survey, no advance information is available about the population. While the generation of random numbers (RDD) will provide a defensible sampling strategy, the randomness alone will be insufficient to insure that the final sample is actually representative of the jurisdiction's population. Nonrepresentativeness may occur either in the original sampling frame if, for example, low-income persons or minorities are less likely to have phones, or in the non-responses or refusals if victimized persons or some other group are less likely to participate in the survey. The resulting level of error in the survey estimates will depend on how many persons are missed and how different they are from those that you are able to interview. Missing a large portion of a distinctive subgroup in the population can greatly reduce the usefulness of the survey data. For most police agencies, some demographic characteristics of citizens are of critical importance in collecting information. The exposure of citizens to police and their opinions of police, as well as victimization experience, vary based on demographic and neighborhood characteristics. Thus information about these characteristics must be collected during any survey and must be monitored as the survey is being conducted to make sure that enough respondents with those characteristics are being surveyed. For most police agencies, these relevant characteristics may include gender, age, race, and income level. Additional variables may be critical in a particular jurisdiction. Once the survey is administered and demographic characteristics of the respondents are developed, these findings can be compared with census data for your jurisdiction to ensure that respondents are representative. For groups which may be underrepresented in the final sample, the responses may be weighted to increase their relative proportion for the entire sample. Survey administration Carrying out a survey Conducting a large survey is a substantial task. This section of the guide discusses the highlights of survey administration. You may want to refer to the appendix for additional resources about this step of the survey. Resources must be identified and capacity for conducting the survey must be assessed. Cities or police agencies often contract with local colleges, universities or research organizations to conduct surveys. These organizations have the resources in terms of telephones, computers and trained personnel to carry out the survey. While this approach may be expensive, this may be the most efficient way to conduct the research. In addition, many police and city leaders feel that there is increased legitimacy attached to research findings conducted external to the police department. The perceived legitimacy associated with survey research is not a minor issue. Conducting their own survey of the community may open the city or police department to criticism or make results suspect. However, if results are designed primarily for use within the police agency for example, feedback for allocating police resources or providing annual evaluation information -- an in-house survey may be appropriate. Using the CVS software companion to this guide relieves some of the criticism applied to police surveys. The CVS is a standardized survey which builds on the National Crime Victimization Survey, conducted for over 25 years by the U.S. Census Bureau for the Bureau of Justice Statistics. The community policing questions in the CVS were rigorously field tested by the U.S. Department of Justice in 12 jurisdictions. The validation processes carried out have standardized the survey instrument, so that the same survey will likely be used by other police agencies in the nation. The survey instrument has a great deal of validity on its own merits. Effort and care must be expended to set up and carry out the survey. Resources such as telephones and computers must be allocated and located. Personnel must be selected and thoroughly trained. Methods of supervision and quality control must be developed to insure the accuracy of the surveys, and issues such as time frames must be established. Coordination and quality control Administration of a telephone survey is not complicated but requires careful planning and close adherence to a series of quality control procedures. Planning a survey also involves estimating the time needed to carry out the survey in a relatively compressed period of time. Surveys which are allowed to drag out for months and months run the risk of a major crime event occurring which will alter respondents' perceptions of crime or police in the midst of the survey. To avoid this situation, every effort should be made to conduct the survey as quickly as possible. Estimations can be made of the amount of time necessary to carry out a survey based on the desired sample size. The national CVS requires approximately 10 minutes to administer. This figure, of course, varies from one individual to the next while the Community Policing section will add approximately 5-10 minutes to the interview time. The number of surveys to be conducted will be determined through the sampling procedure (see above), guiding survey administrators in estimating the amount of staff necessary to conduct the survey. Estimates of the time to conduct a community survey should follow a formula which includes: The number of respondents/cases desired -- e.g., 600 Estimation of refusal rates (30 percent of 12-city CVS) Estimates of eligible respondents (that is, ineligible numbers are screened out -- perhaps around 60 percent) Total number of telephone numbers to be generated and called In this example seeking 600 respondents, 2400 numbers will be generated; 60 percent or 1,440 will be ineligible while 960 will be eligible; and 30 percent or 288 will refuse while 70 percent or 672 will agree to the interview. As general rule, based on the sample size for the 12-city field test of the CVS questions, telephone numbers should be generated for approximately four times the number of surveys desired. This ratio incorporates both ineligibles and refusals. Selection, training, and supervision The selection and training of interview staff and their supervision are critical to administering the survey. All interviewers should be well-supervised. Survey administration should be monitored on a random basis and quality control checks, such as making call-backs to respondents should be carried out to ensure the accuracy of interviews. Training of interviewers should be carried out to ensure that all surveys are conducted as similarly as possible. In general, interviewers should be thoroughly trained to be neutral, precise, and thorough in administering each survey. Training should also emphasize the confidential nature of information being collected. Interviewers should be aware that respondents may be suspicious of providing personal or sensitive information over the telephone particularly about their experience with victimization. Evidence with telephone surveys suggests that these surveys are typically less threatening to respondents than face-to-face surveys. Nonetheless, interviewers should fully brief prospective respondents about the purposes of the survey and the identity of the survey organization. Data management practices should be established to maintain the anonymity of the respondent and the confidentiality of all information collected through the interview. The respondent should be fully assured of anonymity associated with his or her individual experience and opinions. Good interviewers who are able to address respondents' concerns about anonymity may lower refusal rates. Consistent with the practices of the U.S. Census Bureau in carrying out surveys, telephone interviews should adhere to the following practices: * Be friendly and businesslike * Assure the respondent of confidentiality * Refrain from expressing (via words or tone) a personal opinion about answers received -- any comments should be neutral * Pace the interview and always avoid rushing the interview C surveys can be rescheduled for completion at a later time, if necessary * Ask all questions in a deliberate and objective way -- asking no additional questions * Ask all questions in order * Ask all questions exactly as worded -- even if the respondent has previously provided the information * Avoid leading respondents by changing words * Read each question clearly * As instructed, follow up questions with "Any other way?" or other prompts to elicit additional responses * Probe, as necessary -- stating "yes, I see" or repeating questions as necessary may encourage a more complete response to questions. Probes are especially useful if the respondent says "I don't know" * Use neutral probes such as "I don't understand" or "Can you explain that?" * Close the interview by thanking the respondent(s) for their time. Compliance with these basic requirements for conducting telephone surveys should be assessed through routine monitoring of survey administration. As a technique of quality control, all surveyors should be monitored periodically to ensure that the survey is being conducted as designed. Failure to comply with these basic administration requirements may invalidate the survey findings. Analyzing and interperting survey results Analyzing and interpreting survey results is a step of survey research that makes many lay persons panic. To help ease the pain of analysis, the CVS has developed some canned analysis packages to report survey findings. Frequencies and crosstabulations Frequencies report how many persons (and what percent of respondents) responded a certain way to each question in the survey. This is the most basic output from surveys and is a step taken by every survey analyst. For some folks, this may even be the last piece of analysis done on the survey findings! The following table is a frequency table reporting citizen attitudes about drug problems. Concerns about drugs Agree Neutral Disagree Total Drug sales and use in my 61.02% 23.73% 15.25% 100% neighborhood are problems (288) (112) (72) (472) The numbers in the table show the rare situation in which all respondents answered the question. Typically, in a survey with many respondents, each question will have at least a few respondents who do not answer. The number of non-responses is often shown separately, particularly if it is large, and the percents are of the people who did answer the question. While this type of table is interesting, frequencies are most useful for pointing to directions for further analysis. Wouldn't it be useful to know what kinds of citizens are most concerned about drug problems? Crosstabulations compare categories of responses on a particular characteristic. Demographic characteristics frequently influence the way results are interpreted. Concerns about drugs, by age of respondent Drug sales and use in my neighborhood are problems Agree Neutral Disagree Total Age 18-29 21% 18% 61% 100% (30) (26) (87) (143) Age 30-59 43% 21% 36% 100% (74) (36) (62) (172) Age 60 or older 80% 11% 9% 100% (126) (17) (14) (157) Totals 100% 100% 100% 100% (288) (112) (72) (472) This type of table is available as a standardized report with the CVS software. Tables can be generated by the demographic variables such as age, gender and race. The technical manual provides guidance on how to use the software to carry out these tasks. More sophisticated analyses Frequencies and crosstabulations tell most police agencies much of what they want to know from community surveys. We have not discussed significance testing, the criteria for determining the likelihood that an observed relationship is due to chance. You may want to evaluate significance of findings to establish support for an observed relationship. The mathematical formula used to estimate significance depends on a host of factors, including the size of the sample or sub-sample. In addition to frequencies and crosstabulations, more sophisticated analysis may be used by jurisdictions with the interest and ability to do so. For example, surveys which opt to collect location information from residents (such as zipcode or neighborhood) may be able to use GIS systems to map response categories. Analyses should also be framed so that replication of findings from year to year will generate results which provide meaningful comparisons over time. Conclusion Surveys of the public can provide valuable information to police managers and law enforcement policy makers. There are some types of information, such as citizen attitudes and the volume of unreported crime, that cannot be gathered in any other way. This guide has introduced the basic concepts and procedures needed to undertake a community survey. The survey software provided by COPS and BJS incorporates some of the best practices available, thus making the process simpler. Nevertheless, a competent and valid survey, particularly of rare events such as criminal victimization, is a difficult and resource-intensive undertaking. If the sample is too small, there can be little confidence in the results even if it is a random sample. If an important segment of the population refuses to answer the questions or cannot be reached by telephone then making population estimates from your sample is highly problematic. The software and this guide focus on simple random sampling which is appropriate for telephone samples in which nothing is known about the respondents. For many circumstances, simple random sampling is more expensive than other more complex sampling strategies. We strongly recommend that you discuss the substantive problem you want to investigate with a trained survey researcher before starting your project. Appendix A. Terminology CVS -- Crime Victimization Survey -- This is the two-part survey provided by BJS/COPS. It includes the community policing section and replicates the questions contained in the National Crime Victimization Survey conducted by the Bureau of the Census. NCVS -- The National Crime Victimization Survey is the victimization survey conducted by the Bureau of the Census. Parameter -- characteristics of the population of interest, e.g., one gender or one race with victimization experience. Population -- the group in which you are interested. RDD -- Random Digit Dialing -- This is a method for generating a random list of telephone numbers for carrying out the survey. The Windows-based CVS software uses this method to generate a list of numbers for calls. Sample C- a subset or portion of the group you're interested in. Sampling frame -- a list of all possible cases from which you select your sample. Significance testing -- the criteria for determining the likelihood that an observed relationship is due to chance. Statistic -- the estimate of a population parameter from your sample. Probability sample -- a sample in which everyone selected had a known chance of being selected; often probability samples reflect an equal probability of selection. Appendix B. References For further reading on survey research, especially telephone surveys, see -- Babbie, Earl (1999). The Basics of Social Research, 8th edition. Belmont, CA: Wadsworth Publishing. Czaja, Ronald and Johnny Blair (1996). Designing Surveys: A Guide to Decisions and Procedures. Thousand Oaks, CA: Pine Forge Press. Eck, John E. and Nancy G. LaVigne (1994). Using Research: A Primer for Law Enforcement Managers. Washington, DC: Police Executive Research Forum. Lavrakas, Paul (1993). Telephone Survey Methods: Sampling, Selection, and Supervision. Newbury Park, CA: Sage Publications. Lipsey, Mark W. (1990). Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage Publications. Maisel, Richard and Caroline Hodges Persell (1995). How Sampling Works. Thousand Oaks, CA: Pine Forge Press. Saris, Willem E. (1991). Computer-Assisted Interviewing. Newbury Park, CA: Sage Publications. Schutt, Russell K. (1999). Investigating the Social World: The Process and Practice of Research. Thousand Oaks, CA: Pine Forge Press. Webb, Kenneth and Harry P. Hatry (1973). Obtaining Citizen Feedback: The Application of Citizen Surveys to Local Governments. Washington, DC: The Urban Institute. Appendix C. Seven basic steps for conducting telephone surveys 1. Think about and set goals. Take time to be exceptionally clear about what you really want to know from a survey. Modify the survey questionnaire based on your specific objectives. 2. Select a sampling frame by generating telephone numbers with RDD software. 3. Determine the sample size and the sampling method based on your goals.* Select the sample -- that is, select who will be called. 4. Set up the survey administration. Select interviewers such as students or volunteers or contract for administration. Find resources, including telephones, computers and space. Develop data management procedures which secure the data and protect the anonymity of respondents. 5. Administer the survey. Make telephone calls and carry out quality control steps. 6. Analyze the survey data. 7. Report the findings. *You may need some assistance with this step. You can contact a local college, university, or research firm for assistance. Appendix D. Costs Telephone surveys are the least expensive way to get reliable survey information. But there are costs involved. Costs vary from one place to another, but you should be able to predict costs by examining the following cost factors -- Personnel -- telephone callers and supervisors. To save money, some police agencies use volunteers such as students or seniors or other community members. The number of personnel you need is based, of course, on the sample size you select and the time necessary to conduct a single survey. You will also need personnel to be involved in planning the survey, coordinating survey tasks, analysis, and report preparation. Training -- Training of callers and supervisors will require time and staff, and materials. Telephones -- Each caller needs a telephone. Calls can be scheduled for evening hours, thus permitting the use of telephones which may be otherwise used during daytime hours. Space -- Sufficient space is necessary for callers and their necessary equipment. You may have space available within the police agency or elsewhere in local government. For at least a portion of the calls, evening or weekend access to buildings will be necessary. Computers -- Windows-based PCs for recording survey information. One computer per caller. Software -- There is a small cost associated with the survey software. You will also need software such as SPSS or SAS for any sophisticated analysis. Although the CVS includes some standardized report features, these are limited. Contractual services -- You should include costs for any assistance you may need with sampling, making calls, or analyzing data. These costs may vary from enough to carry out the entire survey task to nothing, if you have the expertise in-house. END OF FILE 10/15/99 PM