A Standard Scoring System for Impact in the Social Sector

A Standard Scoring System for Impact in the Social Sector
Burden Utility Weights Sample and Use Case

This article provides samples of the data that Effective Give has been collecting through our Daily Good Program. This data can be used to create a universal scoring system for the social sector, making it possible to compare impact across non-profit cause areas. We believe that this data has the potential to increase the impact produced from the social sector by 10x over the next decade. 

Important disclaimer: This is sample data only. We are not publishing final data for these burdens because we have not hit our target margin of error for each data point (we need a higher sample size to achieve this). We wanted to share this information to start a conversation and see what we might learn from those seeing it for the first time. Please consider joining the conversation by providing feedback in the comments. 

Data Collection 
The data was collected using surveys from Effective Give’s Daily Good program. Participants receive surveys either via email or a mobile app. Each survey represents a single burden that causes suffering in our society and participants are asked the same three questions for each burden. Participants are prevented from taking surveys more than once. 
In this sample we will be reviewing data from five burdens: 

1. Localized Bone Cancer 
2. Fur Farming 
3. Distant Lung Cancer 
4. Alcohol Use Disorder 
5. Localized Stomach Cancer 

The descriptions and common symptoms that we use for each of these burdens is captured in the index at the end of this article. 

Question Structures
We use three separate questions structures to gather data for each burden. These are:

1. Standard Gamble: This structure is designed to capture the upper bounds of the most severe burdens by asking participants to gamble between life/death for the chance to resolve each burden for 100 of those currently affected by it. 
2. Visual Analog Scale: This structure is the simplest for users to understand and helps to capture the lower bound of burdens that might not be captured in the other two question types. 
3. Time Trade-Off: This structure helps to capture a user’s propensity to “give something up” to help resolve a burden. It can be particularly useful for burdens that are not severe enough to warrant a gamble on one’s life but would warrant some sort of sacrifice which might not be captured in the visual analog scale question structure.

Analyzing the Data 
The first three charts (black) capture the data for each burden from each of the three question structures. One of the things that we are continuing to analyze is how each question type impacts the answers for each burden to identify potential trends, issues, and benefits. Ideally, we would be able to limit our survey to a single question to capture an accurate representation of how society values each burden. However, we need to test whether this is viable and whether it negatively (or positively) impacts that “accuracy” of the answers. 

The final chart provides an aggregate of the means (averages) from each of the previous three graphs along with the range of those means. 

All graphs show the range of data, which is indicated by the line for each burden, and the mean of the results which is indicated by the dot.

Data Use Case – The “Scoreboard”
The vision behind this project is to establish a comprehensive database of these burden utility weights. This is the key data that is preventing the social sector from having a universal metric that can be used to measure impact across and within cause areas. Assuming this data had a low enough margin of error, it currently does not, we could use it to assign a “score” each time a burden is averted or resolved by a charity. Combining this data with other existing data such as cost per outcome would very quickly help individual donors, foundations, and charities to understand where they stand in terms of impact using a single metric.

Understanding this impact is essential for truly moving the needle for impact in the social sector. We would contest this is far more effective than a money moved strategy (i.e. advocating that funders move money to the highest impact program/s) because it creates room for funders to optimize for impact within the confines of their existing bias framework (such as location bias or cause bias), which is not something that a moved money strategy does well. The rigidity of a move money strategy is what alienates the vast majority of donors that are not ready to go “all-in” with using logic, reason, data to dictate their funding decisions.

What are your thoughts? Is this data needed? Does the social sector need a common impact metric? Where should we prioritize this project relative to other current programs?

We need more participants in the Daily Good program to produce statistically significant results. If you are willing to sacrifice 30 seconds per day to contribute to this data please visit https://www.effectivegive.org/daily-good.html to sign up. 

Index 
Burden description and common symptoms used during data collection:

1. Localized Bone Cancer
   1. Description: Cancer is a disease caused by an uncontrolled division of abnormal cells in a part of the body. Bone cancer is very rare in adults. It starts in the cells that make up the bone. Cancer starts when cells begin to grow out of control. Cells in nearly any part of the body can become cancer, and can spread to other parts of the body. Localized cancer is limited to the place where it started, with no sign that it has spread. There is a 91% 5-year survival rate for localized bone cancer
   2. Common Symptoms: Bone pain, swelling and tenderness near affected area, weakened bones, fatigue, unintended weight loss.
2. Fur Farming 
   
   1. Description: Animal Rights refers to the rights of animals, claimed on ethical grounds, to the same humane treatment and protection from exploitation and abuse that are accorded to humans. The killing or use of non human animals or animal parts for the purpose of manufacturing clothing. Fur Farming is the process of breeding or raising certain types of animals for fur, especially mink, chinchilla, fox, dog, cat, and rabbit. The fur can be obtained after killing or slaughtering an animal.
      
   2. Common Symptoms: Death with potential for suffering prior to slaughter.
3. Distant Lung Cancer 
   1. Description: Cancer is a disease caused by an uncontrolled division of abnormal cells in a part of the body. Lung cancer or bronchogenic carcinoma refers to tumors originating in the lung parenchyma or within bronchi. It is one of the leading causes of cancer-related deaths in the United States. Distant cancer has spread to distant parts of the body. There is a 5% 5-year survival rate for distant lung cancer.
      
   2. Common Symptoms: Persistent cough, coughing up blood, shortness of breath, chest pain, hoarseness, losing weight without trying, bone pain, headache.
      
4. Alcohol Use Disorder 
   1. Description: Chemical Addition refers to addiction that involves the use of substances. Alcohol use disorder (which includes a level that's sometimes called alcoholism) is a pattern of alcohol use that involves problems controlling your drinking, being preoccupied with alcohol, continuing to use alcohol even when it causes problems, having to drink more to get the same effect, or having withdrawal symptoms when you rapidly decrease or stop drinking.
      
   2. Common Symptoms: Unable to limit alcohol consumption, the desire and inability to reduce alcohol consumption, feeling strong craving for alcohol, failure to fulfill obligations due to alcohol use, eliminating or reducing social activities or hobbies, increased tolerance to alcohol, withdrawal symptoms such as nausea, sweating and shaking when alcohol is not consumed.
      
5. Localized Stomach Cancer 
   
   1. Description: Cancer is a disease caused by an uncontrolled division of abnormal cells in a part of the body. Stomach cancer starts in the stomach and is also called gastric (GAS-trick) cancer. It starts when cells in the stomach grow out of control and crowd out normal cells. Localized cancer is limited to the place where it started, with no sign that it has spread. There is a 70% 5-year survival rate for localized stomach cancer.
      
   2. Common Symptoms: Pain or tenderness in abdomen, nausea, blood in stool, anemia.