Summary

In light of increasing interest in and need for evidence of participation at the local level, this article discusses methodological issues associated with measuring social participation and volunteering. Measuring these subjects is not straightforward and the methodology employed in surveys plays a part in influencing the levels of involvement reported. At a national level, surveys of volunteering and the Government’s Citizenship Surveys offer an established way of asking about volunteering. There is much less agreement on how to ask about volunteering at a local level. There is a need to validate findings from local surveys to explore which questions might be the most appropriate for local surveys aiming to measure participation levels.

Introduction

This paper reports on an Economic and Social Research Council (ESRC) funded project which aims to explore methodological issues associated with measuring participation at a local level. The research is timely as local authorities are now required to report on volunteering targets set out in Local Area Agreements (LAAs) and reliable local data is increasingly needed. The research used qualitative interviews to repeat survey questions on participation in organisations and groups with respondents from a previous local survey to test out the questions and to validate survey responses. These interviews provide insight into respondents’ understanding of the questions, and help to highlight possible shortcomings in survey questions aimed at measuring involvement in groups and organisations. Finally, in light of issues raised the paper discusses some concerns for researchers and policy makers.

Context

Over the past decade social participation and volunteering have received considerable political attention. The Labour Government has invested money in a variety of initiatives to promote volunteering (see Low et al., 2007) and regard increasing levels of participation and volunteering as indicators of strong communities. In particular, the Government has promoted community involvement as part of its civil renewal agenda (see Lowndes et al., 2001; Brannan et al., 2006), and encouraged active participation in communities through numerous initiatives across a range of policy areas (Williams, 2003).

Participation and volunteering are also now key targets set by the Government and are included in a set of National Indicators (NIs) which reflect national priority outcomes. The Local Government White Paper Strong and Prosperous Communities (CLG, 2006) introduced a set of 198 National Indicators which are included in the local government performance framework and cover all the national priority outcomes which local authorities are responsible for delivering. Two national indicators in particular are concerned with participation and volunteering – NI 3 ‘civic participation in the local area’ and NI 6 ‘participation in regular volunteering’ (CLG, 2008).

Levels of participation and volunteering are monitored and evaluated primarily through the series of national Citizenship Surveys (see Attwood, 2003; Green et al., 2004; Kitchen et al., 2006). Other large national surveys like the General Household Survey (GHS) and a number of national surveys of volunteering also measure the prevalence of volunteering (see Field and Hedges, 1984; Lynn and Davis Smith, 1991; Davis Smith, 1998; Low et al., 2007). The surveys collect information on a variety of topics including participation in groups, clubs and organisations and volunteering.

The UK does not have a common national definition of volunteering, and various definitions are used in reports, surveys and legislation. Usually definitions state volunteering involves unpaid activity where someone gives their time freely to help an organisation or an individual who they are not related to. Volunteering covers a wide range of activities and can be defined as:

An activity that involves spending time unpaid, doing something that aims to benefit the environment or individuals or groups other than (or in some cases in addition to) close relatives(Home Office, 2005).

Within this general definition there are different types of participation, and volunteering is most often split into informal and formal activity. Formal volunteering is classed as that which takes place within groups, clubs or organisations whereas informal volunteering is undertaken on an individual one-to-one basis to others who are not members of the family. According to Attwood et al., (2003) formal volunteering has its roots in social participation but involves greater commitment. Social participation is being involved in groups, clubs or voluntary organisations. Definitions of volunteering also distinguish the frequency of activity. Regular volunteering is usually defined as that which takes place at least once a month or more; occasional volunteering takes place less frequently than once a month; and one-off acts of volunteering are classed as episodic volunteering.

On the whole, data on participation and volunteering are collected at the national and regional level, and there is much less information available at a local level. Some local information is obtainable following the release of data from the Place Survey in June 2009. Administered by local authorities, the Place Survey will be used to measure a number of NIs set out in the local government performance framework including NI 3 and NI 6. Data will be used by local authorities and other partners to monitor progress on volunteering targets set out in some LAAs. LAAs are at the heart of the performance framework for local authorities and are the means for agreeing shared targets with central government. Each LAA will have up to 35 national priority targets with specific improvement targets set for each.

Although there are different surveys gathering information on levels of participation and volunteering, these topics remain difficult to measure and comparison of figures between different surveys is not straightforward. Research methodology employed in surveys can influence the levels of volunteering and participation reported in studies.  The definition of what is being measured, the technical aspects and the scope of studies all play a part in determining the levels participation and volunteering reported (see Lyonset al., 1998). Even the phrasing and extent of questioning in surveys of voluntary activity have been shown to account for differences in reported levels of participation (see Evans and Saxton, 2005; Lynn, 1997). Lynn’s work which examined 11 surveys of voluntary activity in Britain between 1976 and 1990 concluded that the main reason for the difference in the numbers reporting participation between the studies was the extent of the questioning. The more time and more prompts people were given resulted in higher estimates of participation in voluntary activity. Lynn also believed that responses were higher in surveys that did not use the terms ‘voluntary work’ or ‘voluntary activity’. A body of research from the USA/Canada examining the effect of methodology (see Steinberg et al., 2002; Rooney et al., 2004) and other methodological considerations (see Hall, 2001; O‘Neill, 2001) on reported levels of volunteering also supports such findings.

Despite the methodological problems associated with measuring participation and volunteering, through the series of national surveys of volunteering and the Citizenship Surveys there is a fairly well established way of asking about volunteering at a national level. In the case of formal volunteering, these surveys tend to use a system of extended questioning with prompts to establish participation in a range of different groups, clubs and organisations. Once involvement in any of the groups, clubs or organisations is determined, and in order to assess whether it can be defined as formal volunteering, the surveys ask whether involvement in the particular groups, clubs and organisations picked out has involved any of a wide variety of listed activities which have been defined as volunteering. Surveys which can devote only a limited amount of questionnaire space and interviewer time to the topic use shorter questions. They tend not to ask about participation in a list of different types of groups, clubs and organisations and do not provide examples of activities which are classed as volunteering. The Place Survey for example, employs only one question to establish regular volunteering, does not provide categories of groups, clubs and organisations for respondents to choose from and conflates both formal and informal volunteering into the question.

There is much less agreement on how to ask about volunteering at a local level. One reason is it is not really feasible to repeat national surveys at a local level. Such an exercise would be prohibitively expensive. Instead local surveys have developed in an ad hoc way and have failed to develop consistency when asking about participation and volunteering often resulting in data which is incomparable with either national figures or to other local areas.

Those wanting to collect information at a local level are faced with a number of problems. First of all there is a dearth of reliable information on participation levels at a community level. Secondly, small scale surveys are likely to have limited time and resources to ask a long series of questions with many prompts. Incorporating an extended set of questions on participation and volunteering typically used in established national surveys will be difficult. Thirdly, comparing reported levels of participation and volunteering with other studies will be problematic and at the very least will require an awareness of how the research design, methodology and questioning methods employed in a study may influence results. Finally, limited evidence is available on how to measure participation at a local level in a way which is both methodologically sound and practically feasible. Given these concerns, information and evidence on how well questions have worked in previous local studies is likely to be useful. Research is needed to validate responses from local surveys and to explore which questions might be the most appropriate for local surveys aiming to measure participation levels.

The South Yorkshire Social Capital Survey

The data used in this research was originally collected as part of an earlier longitudinal study which explored social capital in nine ex-coalfield communities in South Yorkshire in 2000 and 2004 and examined the relationship between social capital and health, and social capital and social participation (Green et al., 2000; Gilbertson et al., 2005; Gilbertson and Manning, 2006a; 2006b).

The South Yorkshire Surveys in 2000 and 2004 measured levels of participation in groups and local organisations. The survey used two questions from the social capital module which was being piloted by the Office of National Statistics for use in the next GHS at the time. These questions were chosen for the South Yorkshire study to take advantage of the developmental work around defining and measuring the concept of social capital. Based on the following definition of social capital: ‘networks together with shared norms, values and understandings that facilitate co-operation’ within or among groups’. (Cote and Healy, 2001: 41), the social capital questions built on the typology developed by Blaxter et al.(2001), the GHS and the Citizenship Surveys.  Five dimensions of social capital were identified, one of which was social participation or involvement in groups, clubs and organisations. The full set of harmonised social capital questions (ONS, 2008, version 3), are now incorporated in a number of government surveys including the GHS. Cross governmental harmonisation has standardised a number of definitions, concepts and questions used in the wide range of government surveys to make it easier for users to draw robust comparisons between different data sources.

The first question used in the South Yorkshire Survey was: Have you been involved in any local organisation(s) over the last three years? In 2004, a second question was added to the survey which asked respondents to select from a list of groups any they had taken part in over the last three years. This question aimed to gain a more detailed understanding of participation in these areas. The questions employed in the South Yorkshire Surveys were similar to questions in the GHS in 2000 and 2004/5 (see Figure 1), and despite differences provide the best benchmark data available.

The differences between the GHS questions and those used in South Yorkshire were mainly due to the need for the South Yorkshire Survey to be kept relatively short so as not to alienate an already over-surveyed population. The project also trained and employed residents to undertake interviews and the survey unit responsible advised that inexperienced resident interviewers were likely to be more successful with a simple user friendly questionnaire. By 2004/5 the timeframe of the GHS questions had changed from three years to twelve months. However, in order to maintain consistency and ensure both waves of the South Yorkshire Survey were comparable, it was decided that the timeframe for the questions in the 2004 survey should be kept to three years.

The questions included in the GHS ask about involvement in three types of groups, informal, local and national. The three questions act as probes and although they are repetitive to respondents, this approach increases the number of groups reported (see Green and Fletcher, 2003). The GHS is similar to the Citizenship Surveys and national surveys of volunteering in that they ask respondents to pick out groups, clubs or organisations that they have taken part in over the last 12 months. However crucially, in the 2004/5 GHS, this is preceded by a yes/no filter question: respondents only get asked to pick out types of groups if they have stated ‘yes’ to having taken part in a group. The list of the types of groups is also shorter in the GHS and does not include categories for safety and first aid, justice and human rights, citizens group or two separate categories for children’s education and young people’s and children’s activities outside schools. These differences are likely to result in lower estimates of social participation in the GHS to those reported in either of the other surveys.

Figure 1: Survey Questions

What Activity are the South Yorkshire and General Household Survey Questions Measuring?

Instructions held in the data archive (ONS, 2006) indicate the 2004/5 GHS questions on social participation should be summed to give an overall measure of involvement.  As such, it is feasible they capture some participation which could be classified as formal volunteering too.  Questions in the 2004/5 GHS and the South Yorkshire Survey do not ask about the level of responsibility or commitment involved and, therefore, it is not possible to ascertain what activity reported by respondents could potentially be classified as formal volunteering.  To differentiate volunteering from social participation, the 2004/5 GHS follows the questions outlined in Figure 1 with questions specifically about unpaid help to any groups, clubs and organisations (formal volunteering) as well as questions on help to individuals who do not live with respondents (informal volunteering).  In the GHS the extent of both formal and informal volunteering can then be reported separately to that of participation in groups, clubs and organisations.  Used alone the questions in the South Yorkshire Survey do gather information on a wide range of social participation activities but do not provide any data on the type or quality of the involvement in local organisations or groups.

Levels of Involvement in Groups and Organisations Reported in the South Yorkshire Survey and the General Household Survey

Although the differences in questioning outlined in Figure 1 complicate comparison, there will be some overlap between what the questions employed in the South Yorkshire Survey and those in the GHS 2004/5 are measuring. Levels of involvement in local organisations reported in both surveys are similar and this may indicate that these questions are capturing similar information. In 2004, 23 per cent1 of South Yorkshire respondents reported being involved in a local organisation compared to 24 of 2004/5 GHS who reported involvement in a local or community group, club or organisation. However, when responses to the three questions in the 2004/5 GHS are summed to give an overall measure of involvement and participation levels are compared to those participating in groups in South Yorkshire,  there are significant differences between the two (Figure 2).

Figure 2: Levels of Social participation

Note: * Differences are significant at the 0.05 level; ** differences are significant at the 0.01 level

These variations may well be accounted for by differences in the survey populations, but it is possible the wording and approach to questioning may play a part in explaining some of the difference. The effect of the repetitive questioning in the GHS could contribute to the large difference in the proportion of respondents who report taking part in one or two groups. Uncertainty about whether the second South Yorkshire question captures information measured by all three of the GHS questions also makes a comparison of these figures more difficult to interpret.

Evidence from Qualitative Interviews with Respondents who took part in the South Yorkshire Surveys in 2000 and 2004

Re-examining the South Yorkshire Survey data for the ESRC study has involved 42 semi-structured qualitative interviews with respondents who took part in both surveys in 2000 and 2004. Unless stated otherwise, the results presented below are for 38 interviews which have been analysed in full. The interviews provided an opportunity to repeat the survey questions with respondents in order to ‘test out’ how they worked. Another purpose of the interviews was to discuss with respondents their understanding of the questions and ask them about their responses to the survey questions in 2000 and 2004.

What activities did the questions capture?

Responses to both of the questions employed in the South Yorkshire Survey included a wide range of activities. At the qualitative interview when respondents were asked the first question about involvement in local organisations and activities, answers included helping out at a local cubs group, membership on the committees of a local bowling club, play group and a model engineering society, membership of the local Labour party, being involved in a local National Trust group, being a trustee of a local leisure centre, attending a course at the local Sure Start and setting up a petition (under the activities part of the question). A list of responses is presented in Figure 3 and shows that 18 of 38 respondents said they were involved in a local organisation or activities.

Figure 3: Responses to the South Yorkshire Local Organisation Question

The activities described above can be classified under three of the four types of community participation defined by Attwood et al., 2003 and cover social participation, civic participation and formal volunteering. At least 11 of the responses can be described as formal volunteering and are marked in bold. These activities (apart from setting up a petition) were then included under an appropriate type of group category listed in the second South Yorkshire question.

Although the first question encapsulated a variety of activities it did not pick up any involvement in local social clubs or working men’s clubs, organisations which remained common in the communities surveyed. In six cases respondents who attended social clubs, some of them regularly, did not include the activity under the first question. An explanation can be found in how respondents regarded the question in the next section below. Furthermore, the exclusion of social club activity also seems to reflect how the clubs were largely utilised. Respondents often used the clubs for social drinking and whilst some had been active in the past, or were involved in societies and events organised through the social clubs, a number simply paid their annual subscriptions and visited occasionally. One respondent when asked why he had not included his membership of the local social club in the question commented that it was more of a ‘pleasure thing’ and not something he thought the interviewer was ‘asking about’ or would be ‘interested in’. Another respondent who had been involved in a sports and social club for 12 years and whose participation had included attending events and providing general help for several activities organised by the club, did not regard the social club as local to where she lived, and so, did not include her involvement in the first question. The relevance of the use of ‘local’ in the first question is discussed in the next section of the paper.

How did respondents perceive the question on involvement in local organisations?

A number of respondents viewed the first question as being about greater commitment, more direct and often regular involvement, being active, being one of the organisers, and even perhaps volunteering.

Yeah being involved and ‘erm going every week to whatever meetings they’re having… if you get involved they expect you to go and you’re committing yourself aren’t you, you’re committing.  It’s a level of commitment that you’ve got to be there haven’t you?  You’ve said you’ll go, you’ll go.

It’s probably that what I thought you were asking really was were I involved in organising events, being part of committees etc.

I thought you meant what I helped out at, not what I’ve done generally.

Such a distinction suggests this question implies something different to respondents than the question about groups and it may capture slightly different aspects of participation.  It is also possible the wording of the question influenced responses.  The first question asks about ‘involvement’ rather than ‘taking part’ and appears to suggest a greater degree of commitment to respondents.  There was also a sense that involvement in a local organisation might be about something more ‘formal’ as opposed to just going along to something as part of a group.  To a certain extent this understanding of the question is reflected in the activities respondents included in their answers, with over half of those who said they were involved in a local organisation listing a formal volunteering activity.

Another issue raised by respondents was the ‘local’ aspect to the first question.  A number made a definite assessment of what they regarded as their ‘local’ area and excluded activities they then included in the second question on types of groups.

I still say no again, I’m a member of the golf club, I’ve been a member of that club since 1962 and at that time when I joined it wasn’t part of this area, it were Wath on Dearne, now it’s part of Rotherham Borough, but I just don’t consider, same as going to see Rotherham United play football, it’s not local really.

I wouldn’t say it’s in the local community but nearby.
Is that why you didn’t mention it in the first question as well?
Probably yeah because I don’t think of that as my local community.

When asked about what respondents saw as ‘local’, for many it did not extend beyond a few streets from where they lived or outside of their communities.

they’re not actually part of this community ‘cos the school she goes in isn’t in this area, it’s more or less in the town area’ … I’d place that as more in town’s community rather than our community (respondent has helped on a stall at daughter’s school summer fair and been to a coffee morning at the school which is less than a mile down the road).

It is also worth noting the order of the questions in the South Yorkshire Survey and the use of ‘local’ in the first question but not in the second probably influenced responses.  By ordering the questions in such a way there was a risk the first question could potentially contaminate responses to the second.  Respondents may have thought the second question was asking solely about the local area rather than asking about involvement in any groups they took part in.

Did the South Yorkshire Survey responses reflect levels of involvement?

When asked about their survey responses most respondents felt their answers were a fair reflection of their participation at the time. Given respondents were being asked about previous responses they had made to a survey, it is possible they perceived the interviewer to be checking up on them and felt under some pressure to confirm their answers were correct. Also, the length of time since the surveys could possibly have made it more difficult for respondents to recall what they were doing at the time. Despite these limitations respondents openly discussed their past and present participation and it was possible to build a picture of their involvement over time and assess how well survey responses estimated involvement.

An examination of survey responses coupled with an analysis of qualitative material reveals a picture which is not as straightforward as respondents perhaps thought. As already mentioned the first South Yorkshire question on involvement in local organisations did not pick up membership of local social clubs or working men’s clubs. Given respondents understanding of the question and the overall level of involvement described this is not really surprising. However, in five cases where respondents had answered ‘no’ to the local organisation question their social club membership was not subsequently picked up by the second South Yorkshire question, even though this option was included in the list of groups.

With regard to the second South Yorkshire question, it seems likely the 2004 survey responses under reported the extent of involvement in groups. Table 1 shows the number of groups reported in the survey compared to the estimated number our additional assessment suggests was the case in 2004.

Table 1: Number of Groups Reported

	Number of times chosen in 2004 South Yorkshire Survey	Qualitative interview assessment of position at 2004
Hobbies/social clubs	5	14
Sports/exercise	5	8
Adult education	4	7
Local community/neighbourhood	5	5
Children/young people	5	5
Trade Union	1	5
Religious	4	4
Older people	2	4
Health, disability, welfare	2	3
Political	1	2
Environmental	1	1
Other	1	1

The category with the largest discrepancy is hobbies and social clubs followed by trade unions but some of the groups only show a small difference or none at all. It is worth mentioning the under reporting of trade unions. Subscription to a trade union alone would not be counted in the GHS 2004/5 questions. Instructions for interviewers indicate that simply paying a subscription or being a member of a group but not interacting with other members does not qualify (Green and Fletcher, 2003). The extra South Yorkshire cases fall into this category whereas the one case which the 2004 survey originally picked up was extremely active in trade unions.

Of 36 respondents, our analysis overall suggests the survey responses for 19 were accurate and 12 were inaccurate. In five cases it was not possible to make an assessment since information was either unclear or missing from the survey. Of the 12 incorrect cases, four did not pick out any groups and eight were involved in other activities which were not picked up by the second question. Of these eight, half did not mention trade union membership.

Conclusions and Concerns for Researchers and Policy Makers

The evidence presented discusses some difficulties associated with measuring volunteering and participation generally. It is widely recognised that measuring these subjects is not straightforward and the methodology employed in surveys plays a part in influencing the levels of involvement reported. As a result cross survey comparisons can be problematic. Interpreting data from different surveys and sources needs an awareness and understanding of how different approaches and variations in questioning techniques may influence responses.

Researchers embarking on work in this field at a local level need to carefully consider the robustness of the questioning methods employed to measure participation, and how questions from larger scale studies are incorporated into local surveys. Our comparison of questions measuring levels of participation in groups and organisations in the GHS and adapted in the South Yorkshire Survey, indicates how even small differences in the questioning methods and wording employed influences the data collected and affects data comparability. Our qualitative interviews suggest there may be under reporting of groups in some categories.

In addition evidence from qualitative interviews emphasises the importance of considering respondents understanding of the questions and the context of local communities in which surveys are being undertaken. Whilst our findings may not be surprising given the legacy of close knit mining communities in the study areas, they do illustrate a particular issue about asking questions which relate to a local area and whether this area should be defined for respondents or not. The Place Survey and the Citizenship Surveys, for example, ask respondents to consider their local area to be the area within 15 to 20 minutes walking distance from their home. Such instructions will be designed to standardise responses, increase comparability and to aid comparison across other surveys which use similar definitions, but it is possible they impose a definition of ‘local’ on respondents which may not correspond to what they regard as their local area.

The UK Government’s success in promoting volunteering and participation is often expressed in terms of the level of volunteering and active participation citizens are involved in. As such the methodological concerns related to the measurement of participation and volunteering are important since they have a bearing on how progress towards a key policy target is interpreted. A recent briefing by nfpSynergy criticises the Government’s definitions of volunteering and civic participation employed in the Citizenship Survey and argues that they over estimate levels of volunteering (Saxton and Baker, 2009). Whether their claims are well founded or not, debate around the boundaries of what constitutes volunteering raises questions about what is actually being measured and muddies the water when it comes to assessing how effective investment aimed at increasing levels of volunteering may be.

Locally, the lack of reliable data on volunteering makes it difficult for policy makers and other local partners to plan strategically, to monitor progress towards targets and to assess the impact of policies aimed at developing volunteering. Whilst the Place Survey will help to standardise data collection across local authorities and will help to provide comparable data on volunteering at a local level, there are problems associated with it. The data it collects will not be directly comparable to those provided by the Citizenship Survey. The Place Survey employs a much shorter question which is likely to result in lower levels of reported volunteering than those in the Citizenship Survey. It will be difficult to accurately benchmark local performance on volunteering targets in LAAs against the national and regional picture. Baselines for performance targets in LAAs will also be set by the first wave of the Place Survey. It is likely that targets concerned with volunteering will be based on data which do not capture the full extent of participation and volunteering which is already taking place locally.