A Standard Scoring System for Impact in the Social Sector Burden Weights Sample and Use Case
This article provides samples of the data that Effective Give has been collecting through our Daily Good Program. This data can be used to create a universal scoring system for the social sector, making it possible to compare impact across non-profit cause areas. We believe that this data has the potential to increase the impact produced from the social sector by 10x over the next decade.
Important disclaimer: This is sample data only. We are not publishing final data for these burdens because we have not hit our target margin of error for each data point (we need a higher sample size to achieve this). We wanted to share this information to start a conversation and see what we might learn from those seeing it for the first time. Please consider joining the conversation by providing feedback in the comments.
Data Collection The data was collected using surveys from Effective Give’s Daily Good program. Participants receive surveys either via email or a mobile app. Each survey represents a single burden that causes suffering in our society and participants are asked the same three questions for each burden. Participants are prevented from taking surveys more than once. In this sample we will be reviewing data from five burdens:
Localized Bone Cancer
Distant Lung Cancer
Alcohol Use Disorder
Localized Stomach Cancer
The descriptions and common symptoms that we use for each of these burdens is captured in the index at the end of this article.
Question Structures We use three separate questions structures to gather data for each burden. These are:
Standard Gamble: This structure is designed to capture the upper bounds of the most severe burdens by asking participants to gamble between life/death for the chance to resolve each burden for 100 of those currently affected by it.
Visual Analog Scale: This structure is the simplest for users to understand and helps to capture the lower bound of burdens that might not be captured in the other two question types.
Time Trade-Off: This structure helps to capture a user’s propensity to “give something up” to help resolve a burden. It can be particularly useful for burdens that are not severe enough to warrant a gamble on one’s life but would warrant some sort of sacrifice which might not be captured in the visual analog scale question structure.
Analyzing the Data The first three charts (black) capture the data for each burden from each of the three question structures. One of the things that we are continuing to analyze is how each question type impacts the answers for each burden to identify potential trends, issues, and benefits. Ideally, we would be able to limit our survey to a single question to capture an accurate representation of how society values each burden. However, we need to test whether this is viable and whether it negatively (or positively) impacts that “accuracy” of the answers.
The final chart provides an aggregate of the means (averages) from each of the previous three graphs along with the range of those means.
All graphs show the range of data, which is indicated by the line for each burden, and the mean of the results which is indicated by the dot.
Data Use Case – The “Scoreboard” The vision behind this project is to establish a comprehensive database of these burden weights. This is the key data that is preventing the social sector from having a universal metric that can be used to measure impact across and within cause areas. Assuming this data had a low enough margin of error, it currently does not, we could use it to assign a “score” each time a burden is averted or resolved by a charity. Combining this data with other existing data such as cost per outcome would very quickly help individual donors, foundations, and charities to understand where they stand in terms of impact using a single metric.
Understanding this impact is essential for truly moving the needle for impact in the social sector. We would contest this is far more effective than a money moved strategy (i.e. advocating that funders move money to the highest impact program/s) because it creates room for funders to optimize for impact within the confines of their existing bias framework (such as location bias or cause bias), which is not something that a moved money strategy does well. The rigidity of a move money strategy is what alienates the vast majority of donors that are not ready to go “all-in” with using logic, reason, data to dictate their funding decisions.
What are your thoughts? Is this data needed? Does the social sector need a common impact metric? Where should we prioritize this project relative to other current programs?
We need more participants in the Daily Good program to produce statistically significant results. If you are willing to sacrifice 30 seconds per day to contribute to this data please visit https://www.effectivegive.org/daily-good.html to sign up.
Index Burden description and common symptoms used during data collection:
Localized Bone Cancer
Description: Cancer is a disease caused by an uncontrolled division of abnormal cells in a part of the body. Bone cancer is very rare in adults. It starts in the cells that make up the bone. Cancer starts when cells begin to grow out of control. Cells in nearly any part of the body can become cancer, and can spread to other parts of the body. Localized cancer is limited to the place where it started, with no sign that it has spread. There is a 91% 5-year survival rate for localized bone cancer
Common Symptoms: Bone pain, swelling and tenderness near affected area, weakened bones, fatigue, unintended weight loss.
Description: Animal Rights refers to the rights of animals, claimed on ethical grounds, to the same humane treatment and protection from exploitation and abuse that are accorded to humans. The killing or use of non human animals or animal parts for the purpose of manufacturing clothing. Fur Farming is the process of breeding or raising certain types of animals for fur, especially mink, chinchilla, fox, dog, cat, and rabbit. The fur can be obtained after killing or slaughtering an animal.
Common Symptoms: Death with potential for suffering prior to slaughter.
Distant Lung Cancer
Description: Cancer is a disease caused by an uncontrolled division of abnormal cells in a part of the body. Lung cancer or bronchogenic carcinoma refers to tumors originating in the lung parenchyma or within bronchi. It is one of the leading causes of cancer-related deaths in the United States. Distant cancer has spread to distant parts of the body. There is a 5% 5-year survival rate for distant lung cancer.
Common Symptoms: Persistent cough, coughing up blood, shortness of breath, chest pain, hoarseness, losing weight without trying, bone pain, headache.
Alcohol Use Disorder
Description: Chemical Addition refers to addiction that involves the use of substances. Alcohol use disorder (which includes a level that's sometimes called alcoholism) is a pattern of alcohol use that involves problems controlling your drinking, being preoccupied with alcohol, continuing to use alcohol even when it causes problems, having to drink more to get the same effect, or having withdrawal symptoms when you rapidly decrease or stop drinking.
Common Symptoms: Unable to limit alcohol consumption, the desire and inability to reduce alcohol consumption, feeling strong craving for alcohol, failure to fulfill obligations due to alcohol use, eliminating or reducing social activities or hobbies, increased tolerance to alcohol, withdrawal symptoms such as nausea, sweating and shaking when alcohol is not consumed.
Localized Stomach Cancer
Description: Cancer is a disease caused by an uncontrolled division of abnormal cells in a part of the body. Stomach cancer starts in the stomach and is also called gastric (GAS-trick) cancer. It starts when cells in the stomach grow out of control and crowd out normal cells. Localized cancer is limited to the place where it started, with no sign that it has spread. There is a 70% 5-year survival rate for localized stomach cancer.
Common Symptoms: Pain or tenderness in abdomen, nausea, blood in stool, anemia.
Much of the work that we focus on at Effective Give is inspired by a growing movement called Effective Altruism. I first stumbled into this community in 2013 after reading The Life You Could Save written by Peter Singer – a moral philosopher and Professor of Bioethics at Princeton University. Singer is not the founder of the EA movement, but he was, and continues to be, an influential supporter of the core tenants of Effective Altruism. I can still clearly recall a moral dilemma from the first chapter of the book that functioned as a trailhead to the journey of exploring how we can help better the world.
On your way to work, you pass a small pond. On hot days, children sometimes play in the pond, which is only about knee-deep. The weather’s cool today, though, and the hour is early, so you are surprised to see a child splashing about in the pond. As you get closer, you see that it is a very young child, just a toddler, who is flailing about, unable to stay upright or walk out of the pond. You look for the parents or babysitter, but there is no one else around. The child is unable to keep his head above the water for more than a few seconds at a time. If you don’t wade in and pull him out, he seems likely to drown. Wading in is easy and safe, but you will ruin the new shoes you bought only a few days ago, and get your suit wet and muddy. By the time you hand the child over to someone responsible for him, and change your clothes, you’ll be late for work. What should you do? – Peter Singer, The Life You Can Save
I am sure that your response is as simple as it seemed to be for me the first time I read this passage. Who cares about the shoes, the suit, or being late for work – given the opportunity we are jumping in to help the child. Then Singer hits us with a ton of bricks…
10 million children under the age of 5 were dying per year for causes related to poverty. An overwhelming number of these deaths are completely preventable with low-cost interventions that are readily available in most developed nations. While this data has improved marginally since the publishing of Singer’s book, we are still facing these very tragic statistics globally. A point that Singer goes on to make is that nearly all of us could save the lives of others with fairly little effort or sacrifice. In answering the hypothetical dilemma—where we can see the child drowning—most of us state that we are willing to help, but contrasted with reality, most are not taking the necessary steps needed to save the proverbial drowning child.
At the time I was reading Singer’s book I was an active donor, fundraiser, volunteer, and was in the process of launching a non-profit. However, I was not yet thinking about philanthropy in the context of optimizing impact. I was not very strategic, and I was generally responding to the most obvious opportunities. I was reactive. I always felt that my heart was in the right place, but never realized until reading more about Effective Altruism that my head needed to be there as well. There is no doubt that many would-be beneficiaries missed out because I was not yet focusing my volunteer and fundraising efforts with adequate intention.
Truly effective philanthropy requires a balance of both the head and the heart. We must be both empathetic and strategic. That is the foundation of Effective Altruism.
Effective Altruism is a philosophy and social movement that advocates using evidence and reasoning to determine the most effective ways to benefit others.1 It is concerned with helping the proverbial drowning child and challenges us to think about the responsibility that we each have be good stewards of our resources for the purpose of contributing to the greater good. We should care about Effective Altruism for the same reasons that we care to make any positive impact. Effective Altruism provides a framework for us to do good, better. It provides us with the opportunity to do the most with both our head and heart to relieve the suffering of others. I have jokingly used an analogy to introduce Effective Altruism to people in the past – Effective Altruism is what Mother Teresa and Albert Einstein would have named their child – something that marries compassion and intellect to produce something truly beautiful.
While we are still working to collect enough data to substantiate our hypothesis, it is our estimation that the U.S. Non-Profit sector is operating at a fraction of its potential in terms of impact. Through better data, greater accountability, and additional resources that help donors measure and optimize impact we can work to close this gap within our lifetime.
Effective Give is a recognized 501(c)(3) non-profit organization that exists to educate and equip exempt organizations and the public with tools, resources, and information for more sustainable and effective charitable giving.