Question Wording Shapes Measures of the Gig Economy
Today’s labor market discussions are full of ambiguous terms: the gig economy, contingent work, alternative arrangements. These terms can mean different things to different people—including the thousands of workers who complete surveys seeking to measure these concepts. The words used on surveys and the prompts given to respondents matter to the way the questions are answered, and the measures the surveys provide. Two recent studies shed more light on the ways in which question wording impacts people’s survey responses—and the understandings of the gig economy we extract from those responses.
In one of these studies, Katharine Abraham, professor at the University of Maryland and former Bureau of Labor Statistics (BLS) commissioner, and Ashley Amaya of RTI International used Amazon Mechanical Turk surveys to assess the extent to which the BLS Current Population Survey (CPS) questions capture informal work activities. The paper, Probing For Informal Work Activity, was released through NBER on August 8.
The CPS, conducted monthly, is the primary source for labor force statistics in the U.S., including the employment and multiple job-holding rates. In their study, Abraham and Amaya asked the usual CPS questions, which rely on words like “job,” and “pay,” and “profit.” They then asked additional questions, like if respondents did “other things to earn money.” Probing uncovered additional work activity that was not captured by the CPS questions for 21.9 percent of respondents, suggesting respondents did not think of some of their work activity as a “job,” but still engaged in it to contribute to their income.
This survey was not intended to be generalizable, or to measure the size of the workforce in the U.S. However, among the sample, additional probing led to an increase in the number of people participating in the workforce, and especially the number of multiple job-holders. The multiple job holding rate among respondents rose from 23.6 to 43.2 percent—almost 20 percentage points. Similar probing on a nationally representative survey could result in parallel increases, though the CPS multiple job-holding rate currently is only about 5 percent. Despite its limited sample, the study indicates that question wording is important for cueing respondents to include informal and alternative work activities in their employment-related responses, and suggests current measures likely do not accurately capture the range of ways in which people work.
Another recent report does seek to measure the size of the gig economy. Gallup, the consulting company known for its public opinion polling, released The Gig Economy and Alternative Work Arrangements, a report on the nature and composition of the gig workforce on August 16. The report is based on a national survey that estimates 29 percent of U.S. workers rely on alternative work arrangements as their primary job, and 36 percent have some connection—primary or supplementary. It identifies five types of alternative work arrangements: independent contractors, online platform workers, contract firm workers, on-call workers, and temporary workers.
These categories are similar to those included on the BLS’ Contingent Worker Supplement, which found 10.1 percent of workers to rely on alternative arrangements as their primary job. The Gallup survey, however, intended to use broader definitions, which likely explain some of the discrepancy between the two. In measuring contract-firm workers, for example, BLS limited their count to those workers who are assigned to one customer and work at that customer’s site, whereas Gallup counted any self-identified contract worker. Gallup also included in their count online-platform workers as part of their count. BLS asked about platform work, but did not tally those responses with their initial count; they plan on releasing that data this fall.
In addition, differences in sampling methods likely account for some of the difference—Gallup relied on an online panel, and then weighted those responses to represent the population, whereas the CWS relied on telephone and in-person surveys with a random sample—a more rigorous approach more likely to reflect the population. However, the importance of the definitions given to categories of work cannot be discounted, and should be considered by those comparing the studies. Given both definitional and methodological differences, Gallup notes that its findings are not meant to be a comparison to other surveys.
Abraham and Amaya’s research and Gallup’s report highlight the significance of survey design in shaping our understandings of the labor market. How questions are written, how respondents think about those questions, and how results are presented need to be as aligned as possible. As work changes, we need a balance between questions that provide comparable data to previous surveys and new questions that incorporate new ways of working. These are difficult questions for researchers to address, but consumers of research have a responsibility, too, to look carefully at the sources they reference. No single survey can answer every question about work today—instead, a range of thoughtful approaches can give a richer picture of the landscape of work and a better understanding of today’s workers and their needs.