Helmets.org

Bicycle Helmet Safety Institute

Consumer-funded, volunteer staff

Helmets Children Promotions Statistics Search


Evaluating Helmet Campaigns


Summary: Campaign evaluations are important to your helmet program and to others who will follow. We are great believers in field observation rather than telephone sampling.


Introduction


One of the worst problems of helmet promotion campaigns is evaluation. Thorough evaluations are necessary for those who come next and need to know about the best practices that produce the best results. But those who have struggled to put together a campaign and see it through are ready to relax, and are already convinced by their experience that they have done good. So the evaluation phase can be neglected, and the lessons learned are lost. Even worse, the people in the next state are denied the ability to promote their campaign or law with hard numbers and demonstrated results. So we have begun to assemble some material so you don't have to reinvent this wheel.

Our approach is firmly oriented toward scientific observation, and toward field observation methods. Anyone who has worked with bicycle riders and surveys knows the futility of asking a rider how far they have ridden recently, or almost anything else about their riding, and expecting an accurate answer. Ask a room full of second-graders if they have bike helmets and if they use them, and a forest of hands will pop up to give you the answer they know you want. Just ask a teacher. The State of Georgia passed a helmet law, and before they passed it they called people and asked if their kids were using helmets. Then after the law was passed they called again and -- surprise! -- more people answered that their kids were obeying the law and using helmets. The only rider surveys we accept are those based on ride logs kept daily by the hardest-core riders, and those are few and far between.

From these assumptions, our bias is toward the field observation method: actual field counts of who is and who is not wearing a helmet. If all other things were equal, careful counts before and after your campaign or before and after your law passed should show whether or not your actions were generating results. All things may not be equal, but at least you have to try.

Don't miss this one!

For starters, we highly recommend the article on Measuring Community Bike Helmet Use Among Children from Public Health Reports (Vol. 116, No. 2, pp.113-121). It details the ins and outs of using various measurement techniques. The authors share our bias toward actual observation to determine helmet use and go on to say that it can be used to evaluate the risk of injury. We don't agree with all of their findings, but If you are doing helmet counts you want to take the time to read this excellent article first.

The first group of materials below comes from a New York State program that produced a helmet evaluation campaign manual in 1990. The original manual was showing its age, so we have replaced it with newer material from the current program contact: The New York State Department of Health, Bureau of Injury Prevention, email: injury@health.state.ny.us Following that is a summation of the points in a recent article by Les Becker et al in Injury Prevention, which discusses how to choose observation points and how to conduct the observations. It is less detailed than the New York materials, but has more information on methods of selecting observation points. Finally, we have added a chart prepared by the Centers for Disease Control summarizing evaluation efforts from several localities.

New York State Material


The New York State Department of Health (NYSDOH), Bureau of Injury Prevention (BIP), Division of Chronic Disease Control and Prevention, supports a statewide bicycle helmet safety project including helmet distributions, education and awareness for the public and medical professionals, and evaluation research. The project has been funded through various grant awards from the National Centers for Disease Control and Prevention (CDC), the New York State Governor's Traffic Safety Committee (GTSC), and the NYS Developmental Disabilities Planning Council (DDPC), as well as from private donations from insurance companies. It is coordinated with statewide legislation requiring helmets for all children under 14 years of age while bicycling or inline skating. The State has developed an evaluation method for this program, which is presented below.


Evaluation Research

In addition to process evaluation, including documentation of the number of helmets distributed, educational presentations conducted, and educational/promotional materials requested, various strategies have been developed to ascertain the effectiveness of the overall project and determine if there has been a change in bicycle helmet ownership and usage. A school-based survey was administered each year between 1989 and 1993 through a sample of Parent Teacher Associations in upstate counties and in a sample of schools in NYC. This survey has documented an increase in ownership and usage of bicycle helmets, most significantly in younger children.

A protocol for conducting an observational study to determine helmet usage has been developed by Bureau of Injury Prevention staff and distributed within NYS and nationally. Pre- and post- law observation studies were conducted statewide to investigate any changes in bicycle helmet usage due to implementation of the bicycle helmet legislation. In addition, the NYSDOH BIP, under cooperative agreement with the CDC, is conducting a bicycle helmet evaluation project to determine which strategy, or combination of strategies, is associated with increasing bicycle helmet usage.


Method for Taking Observations



Observational Study Design

Bicycle helmet observations are conducted in six counties across NYS, three times a year, Spring, Summer and Fall. Each county has two observers who are trained on the proper methods of conducting observations, and each county has three observational sites (school, park, playground, etc.). Observers conduct a combination of stationary and driving observations at the three sites. Most often each observer picks one site each and they share the third site. Stationary observations are usually 30 minutes in duration and driving observations are usually 60 minutes although variations by county/site are possible.

An example of how the observations may be conducted is listed below: 
 
          Site One                 Site Two         Site Three
----------------------------------------------------------------------

Observer 1. 30 min stationary+                     1. 30 min stationary
  A         60 min driving                             + 60 min driving

         2. 30 min stationary +               
            60 min driving                            
                   
----------------------------------------------------------------------

Observer                     1. 30 min stationary  1. 30 min stationary
  B                             + 60 min driving       + 60 min driving

                             2. 30 min stationary               
                                + 60 min driving                     

----------------------------------------------------------------------
Observations are conducted on non-rainy, relatively warm days. For each site, the observers conduct the observations at the same time of day and observe from the same vantage point during the stationary observation and follow the same route for the driving observation (determined ahead of time during site visits).

Bicycle Helmet Observation Form

 
COUNTY:
PLACE OBSERVED:                 Stationary / Driving 
DATE:          TIME: Begin           End 
WEATHER: Overcast / Partly Cloudy / Sunny         Approx. Temp.
OBSERVER'S NAME:
Alone   or  Together    with (name) 

===================================================================

-------------------------------------------------------------------
BOYS                            Helmet               No Helmet
-------------------------------------------------------------------
            |
            |  10+
            |
            |
            ------------------------------------------------------
            |
            | Under 10
            |            
            |
            ------------------------------------------------------

------------------------------------------------------------------
GIRLS 
-------------------------------------------------------------------
            |
            |  10+
            |
            |
            ------------------------------------------------------
            |
            | Under 10
            |            
            |
            ------------------------------------------------------


===================================================================

Straight line = Correctly worn helmet 
Squiggly line = Incorrectly worn helmet 

Top of box = Sure of age 
Bottom of box = Unsure of age


General Guidelines for Driving Observations

Very Important: For safety concerns please identify a person, other than yourself, to drive while you are conducting the observations.

This driver can be different every time and no training is required. The driver will be reimbursed at $8.00 per hour. If a driving observation is conducted in association with a stationary observation and the driver accompanies the observer on a stationary observation, the driver's time IS reimbursable. Please be sure the driver fills out the driver reimbursement form.

Mileage is reimbursable at $0.31 per mile while on the driving route only. Mileage to and from the driving route is not reimbursable at this time. Please be sure the owner of the automobile fills out the mileage reimbursement form.

Plan the driving route ahead of time based on your knowledge of the community. If a driving route has been pre-established, please try to follow this route as closely as possible. Please submit a map with the driving route clearly marked for each driving observation, even if the route did not change. Please put the date you conducted the driving observation on the map.

Plan an overall loop where the route begins and ends at the same place.

Focus on residential areas rather than business districts or rural areas. Observe for the time allotted based on your instruction packet. If you finish the route before the allotted time is over, restart the route and continue until you reach the time limit. Mark on the map where you stopped. A substantial overlap or shortfall may require modifications the next time to lengthen/shorten the route.

Do not observe while on highways/main streets where traffic is traveling greater than 35 mph. If you must travel these roads to get from one residential area to another, stop the clock and resume once you get to the next area.

If you encounter a traffic jam or construction delay, stop the clock and resume once you are clear.

Avoid double counting children whenever possible. If restarting a route, try to notice if you are observing the same children. Do not count those you know you saw in the beginning. Only observe as you drive one-way on a dead end street. Observing while coming back out will count the same children seen on the way in. Stop the clock if it is a very long dead end and resume when you get back to the main route.

Do not observe within a 2-3 minute (4-5 block) radius of the stationary site. This should eliminate observing children who are riding to and from the site and who may be counted while you are doing the stationary observation.

If you have any questions, please contact the New York State Department of Health, Bureau of Injury Prevention, email: injury@health.state.ny.us.

THANK YOU FOR TAKING THE TIME TO REVIEW THESE INSTRUCTIONS. YOUR HELP IS GREATLY APPRECIATED!


Instructions: Bicycle Helmet Observational Study

1. Observations should be completed between the pre-determined dates. If you have trouble completing them by the end of the time period, please let us know.

2. Conduct stationary and/or driving observations for each site based on the specific instructions provided in your observation packet. (Note the date and time started and stopped, and whether it was a stationary or driving observation on the observation forms.)

3. Remember to observe for the specified number of minutes only. Stationary observations are usually 30 minutes. The length of time for the driving observation may vary by county/site. Please refer to your observation packet.

4. If you do not observe any riders, please call us immediately. You may be asked to do another observation if your schedule/weather/time permits. You will be paid if you are asked to do this extra observation.

5. Observe on non-rainy, relatively warm days. (Note the weather on the observation forms.)

6. Observe the same type of day (weekday/weekend) and time of day at each site.

7. When no reliability assessments are being conducted, all observations should be completed separately from the other observer. Please be sure to coordinate schedules so you do not inadvertently show up on the same day. (Check the appropriate 'alone' or 'together' space on the observation forms.)

8. Make one mark per child in one box noting presence or absence of a helmet, gender (boy/girl), age (10 and older or under 10).

9. When noting age, if you are sure about the age of the child, mark in the upper half of the box; if you are NOT sure about age, mark in the bottom half of the box.

10. If the child is wearing a helmet, note whether the helmet is used properly by marking a straight line for a correctly worn helmet and a squiggly line for an incorrectly worn helmet. (Diagram shows a correctly worn helmet as level on the head, not tilted back or forward or to the side, with the chin strap fastened snugly, not hanging down way below the chin.)

11. If the child is not wearing a helmet, the mark will always be a straight line. There is no way for a helmet to be worn incorrectly (squiggly line) if the rider is not wearing one! 12. Please return final site maps (if applicable), observation forms and reimbursement forms in the enclosed pre-addressed postage paid envelopes once all observations are completed. Please keep a photo copy or some other record of the observations in case forms get lost in the mail. Please limit number of pages to 4-5 per envelope due to postal weight regulations.

If you have any questions, please contact the New York State Department of Health, Bureau of Injury Prevention, email: injury@health.state.ny.us.

THANK YOU FOR TAKING THE TIME TO REVIEW THESE INSTRUCTIONS. YOUR HELP IS GREATLY APPRECIATED!


Helmet Observation Training

This is based on slides prepared to go with training conducted by Patricia O'Connor, PhD, consultant to the NY State Department of Health.

Slide 1

Purposes of Research: I

Slide 2

Purposes of Research: II

Slide 3

Essential Characteristics of Research

Slide 4

Research Design

Slide 5

Types of Research

Slide 6

Observational Studies are Scientific when they:

Slide 7

Now to practice and to establish reliability of observation:

Slide 8

Training Observation Form (above)


Additional Forms

The New York State material includes forms for driver reimbursement, mileage reimbursement and observer reimbursement. There is a bicycle helmet application form, and methods of calculating income to determine who qualifies for low-income helmet assistance.


Maryland's Community-Based Approach


Staffers of the Maryland Department of Health published an article in Injury Prevention, 1996; 2:283-285 by Les R. Becker et al, on experience in the State of Maryland gathering numbers before and after their helmet laws were passed. (The numbers will be available eventually.) Their conclusions:

Selecting Sites for Observation

The article details two methods: asking club cyclists where to find the most riders with some map study thrown in, and asking community informants where to find the most young bicycle riders. The second method, using community informants, yielded more cyclists per hour and more observations per dollar spent than asking club cyclists.


Training Observers and Conducting Observations

For the Maryland studies, high school and college students were trained as observers a day or two before their sessions. After about 30 minutes of training they were able to identify and number passing cyclists while recording their age group, gender, race and helmet use on a form. Observations were made from vehicles and from fixed points between 9 AM and 4 PM on summer days. The sites selected by knowledgeable community members generated from 50 to 450 percent more observations of riders in the under-16 age group than those selected using a combination of bicycle club member recommendations and studying maps for such features as parks and playgrounds. The study authors were unable to say if the increased size of the sample made the results more representative of the target group.


Other Evaluations: Results of Some Evaluations of Helmet Laws



This chart was produced by the Centers for Disease Control in Atlanta, a part of the U.S. Public Health Service. It attempts to summarize the data from some evaluations of bicycle helmet laws

Centers for Disease Control
Evaluation of Legislation and Community Programs
to Increase the use of Bicycle Helmets - Selected Locations

Helmet-use rates are for children except for
Victoria, Australia, which included adults.




 
                                                Helmet use Rates 

                                               Pre-        Post-
Location   Years          Program type        Program     Program 
           Evaluated


Victoria,  March 1983-    Community campaign  6 percent   36 percent 
Australia  March 1990

           Comments: Included education, mass media publicity, 
           support by professional associations and community groups, 
           involvement of bicycling groups, and $10 government rebate 
           for helmet purchases.



 d.o.      March 1990     Helmet legislation  36 percent  73 percent
           March 1991     introduced

           Comments: Hospitalizations for bicycle-related
           head injuries also decreased by 37 per cent,



Howard     1990-1991      Helmet legislation,  4 percent  47 percent
County,                    community campaign 
Maryland    
 
            Comments: Activity prompted by bicycling deaths of two
            children. Use determined by observation. School-based
            survey showed 11 percent and 37 per cent. 



Montgomery 1990-1991      Community campaign   8 percent  19 percent
County,
Maryland 

             Comments: Use determined by observation. Use determined
              by school-based survey was 8 per cent and 13 per cent



Baltimore  1990-1991      No specific helmet   19 percent 4 percent
County,                    promotion activities 
Maryland
           Comments: Served as control county. Use determined by
           observation. Use determined by school-based survey was 
           7 percent and 11 percent.

 


Seattle,   1987-1988      Community campaign   5 percent  14 percent
Washington

           Comments: Included education of parents by physicians;
           advertising in newspapers, on television, and on radio; 
           school presentations; and discount coupons for helmets.



d.o.       1980-1990      Community campaign   14 percent 33 percent

           Comments: Follow-up evaluation of bicycle helmet campaign.



d.o.       1990-1993      Community campaign   33 percent 60 percent

           Comments: Follow-up evaluation of bicycle helmet campaign.
           Bicycle-related head Injuries decreased approximately 67
           per cent among children 5-14 years of age who were members
           of a health maintenance organization.



Portland,  1987-1988      No specific helmet   1 percent  4 percent
Oregon                     promotion activities

           Comments: Control community, use determined by observation.




Barrie,    1988-1989      Educational program  0 percent  0 percent
Ontario
Canada

           Comments: Use determined by limited number of observations.
 


d.o.       1980-1989      Educational program  0 percent  22 percent
                           and helmet subsidy
 
           Comments: Use determined by limited number of observations.
Source:
Injury-Control Recommendations: Bicycle Helmets
Morbidity and Mortality Weekly Report, Vol 44, No. RR-l February 17, 1995
Centers for Disease Control and Prevention (CDC)
U. S. Public Health Service, Department of Health and Human Services
Harborview Injury Prevention Center
Rivara, Thompson, Patterson, Thompson

PREVENTION OF BICYCLE-RELATED INJURIES: Helmets, Education, and Legislation Annual Review of Public Health, Vol. 19:293-318 (Volume publication date May 1998) Frederick P. Rivara, Diane C. Thompson, Matthew Q. Patterson, and Robert S. Thompson. Portion of the abstract: "Helmets have been shown to reduce bicycle-related head injuries for cyclists of all ages involved in all types of crashes including those with motor vehicles. Helmet use has been promoted using educational campaigns, helmet subsidies, and legislation. Careful evaluation of these strategies has shown that these interventions increase helmet use and decrease the incidence of bicycle injuries."



Centers for Disease Control
Evaluation of Legislation and Community Programs
to Increase the use of Bicycle Helmets - Selected Locations

Helmet-use rates are for children except for
Victoria, Australia, which included adults.


 
                                               Helmet use Rates
 
                                               Pre-        Post-
Location    Years          Program type      Program      Program
            Evaluated


Victoria,   March 1983-    Community        6 percent  36 percent
Australia   March 1990     campaign   

            Comments: Included education, mass media publicity, 
            support by professional associations and community
            groups, involvement of bicycling groups, and $10
            government rebate for helmet purchases.



Victoria,   March 1990     Helmet         36 percent  73 percent
            March 1991     legislation
                           introduced

            Comments: Hospitalizations for bicycle-related 
            head injuries also decreased by 37 per cent,


Howard      1990-1991      Helmet           4 percent  47 percent
County,                    legislation,
Maryland                   community campaign 
 
             Comments: Activity prompted by bicycling deaths of
             two children. Use determined by observation. A
             school-based survey showed 11 percent and 37 per
             cent. 



Montgomery  1990-1991      Community   8 percent  19 percent    
County,                    campaign 
Maryland 

              Comments: Use determined by observation. Use
              determined by school-based survey was 8 per cent
              and 13 per cent



Baltimore   1990-1991     No specific   19 percent 4 percent 
County,                   helmet promotion  
Maryland                  activities

            Comments: Served as control county. Use determined 
            by observation. Use determined by a school-based
            survey was 7 percent and 11 percent.



Seattle,    1987-1988      Community    5 percent  14 percent   
Washington                 campaign

            Comments: Included education of parents by
            physicians; advertising in newspapers, on 
            television, and on radio; school presentations; 
            and discount coupons for helmets.



Seattle,    1980-1990      Community   14 percent 33 percent
Washington                 campaign 

            Comments: Follow-up evaluation of bicycle helmet
            campaign.



Seattle,       1990-1993      Community   33 percent 60 percent
Washington                    campaign 

            Comments: Follow-up evaluation of bicycle helmet
            campaign. Bicycle-related head injuries decreased
            approximately 67 per cent among children 5-14 years
            of age who were members of a health maintenance
            organization.



Portland,   1987-1988      No specific   1 percent  4 percent
Oregon                     helmet 
                           promotion 
                           activities

            Comments: Control community, use determined by
            observation.



Barrie,     1988-1989      Educational  0 percent  0 percent
Ontario                    program
Canada

            Comments: Use determined by a limited number of
            observations.



Barrie,     1980-1989      Educational   0 percent  22 percent
Ontario                    program and
Canada                     helmet subsidy

            Comments: Use determined by a limited number of
            observations.
Source:
Injury-Control Recommendations: Bicycle Helmets
Morbidity and Mortality Weekly Report, Vol 44, No. RR-l February 17, 1995
Centers for Disease Control and Prevention (CDC)
U. S. Public Health Service, Department of Health and Human Services
This chart is part of the source document, the rest of which
is available from CDC




Swedish Study

Swedish authorities published a paper in 2003 evaluating helmet campaign strategies. The overall conclusion is that a considerable increase in helmet use by cyclists could be achieved by noncompulsory measures, but the use levels are not as high as those achieved by legislation.


Dutch Study

A similar study in the Netherlands turned up some different conclusions about helmet use. We have the abstract and a link to the original study in Dutch.


More Studies


The best list of studies and references on educational campaign evaluations is the page titled "Bicycle Injury Interventions Programs to Increase Helmet Use: Education" now on archive.org and somewhat outdated, but originally from the Harborview Injury Prevention Center site.

In 2016 a study was published titled "Challenges in the accurate surveillance of booster seat and bicycle helmet usage by children: lessons from the field". The abstract says: "..This paper examined the challenges that confront efforts to collect surveillance data relevant to child traffic safety, including observation, interview, and focus group methods. Strategies to address key challenges in order to improve the efficiency and accuracy of surveillance methods were recommended. The potential for new technology to enhance existing surveillance methods was also explored." The article is available free on the International Journal of Environmental Research and Public Health site.

Back to the top

Back to the Home Page