Our Evaluation Framework
One of our values at Effective Altruism Ventures is transparency. Unlike traditional VC firms, we have no proprietary models to protect, and no secret competitive advantages that we must defend. We succeed if we help good projects get funding and don’t help bad projects get funding.
In the spirit of transparency, this document explains how we choose which projects to support. It first discusses the assumptions behind our general approach and then discusses our model for evaluating projects and the process we use to assess projects according to this model.
This model is a draft and will undergo revision over time as we gain more data. Feedback, criticisms, or questions should be directed to kerry@eaventures.org.
How we evaluate projects
Collectively our team has founded 25 organizations and multiple startup accelerators. We use this experience to help design our process.
However, we are also skeptical of unstructured expert judgment. Evidence suggests that in high-complexity, high-uncertainty environments, experts are bad at making overall judgments. (The best summary of this literature is given in sections 3.2-3.6 of Rational Choice in an Uncertain World.) In over a hundred comparisons of raw human judgment to human judgment supplemented with statistical models, raw human judgment has never won. Yet, the evidence also suggests that experts are good at discovering the key variables in their prediction. They are just bad at combining multiple sources of information, ignoring irrelevant factors and coming to an overall judgment.
We merge expert judgment with statistical models of project success. We used our expertise and the expertise of our advisers to determine a set of variables that is likely to be positively correlated with project success. We then utilize a multi-criteria decision analysis framework which provides context-sensitive weightings to several predictive variables. Our framework adjusts the weighting of variables to fit the context of the projects and adjusts the importance of feedback from different evaluators to fit their expertise.
Multi-criteria decision analysis
We evaluate three criteria and 21 sub criteria to determine an overall impact score. The elements we evaluate are detailed below.
Criteria 1: Cause Area
We break the project idea down into two factors. First, we look at the cause area of the project to see if it is a high-impact project. Second we look at the details of the specific project we’re evaluating. Since we focus only on projects that help create a flourishing world, the project idea is one of the first variables we use to reject a project.
For the cause area we apply three tests:
We assess whether the problem being solved is important.
We assess whether the problem space is crowded.
We assess whether the problem being solved is tractable.
Some projects will create a flourishing world by making money and then donating this money to other high-value projects. We are open to this approach and we apply a different model to evaluating these project ideas.
Is the problem important?
The next three questions arise from work done by GiveWell and 80,000 hours. In assessing importance we ask ourselves a few questions:
Will progress on this problem contribute to building a flourishing world?
If more progress is made on this problem how many people will benefit?
Does progress on this problem help develop knowledge and expertise that will be useful in many possible future worlds?
Does working on this problem help us make progress on other causes?
Is the space surrounding the problem crowded?
In general, the less resources being spent on a problem, the greater impact we can have by getting involved. We prefer problems with a mismatch between the importance of the problem and the resources going to solve it.
Some methods of assessing this include:
Is there an argument that society undervalues this area for the wrong reasons?
Is there an argument that this project will be underfunded for the wrong reasons?
Are existing projects already working to solve this problem?
Is there a compelling reason that normal market forces will not solve this problem?
Is the problem tractable?
A problem may be important, but progress on the problem must be attainable. We look for a few attributes:
Do well-defined interventions in this area exist?
Is there a strong theory about why particular interventions should work?
Is there a track record of success in this area or in related areas?
Will the success of this project be measurable? How will it be measured?
For established projects, we use signs of the success of the projects themselves instead of other ways of assessing tractability.
CRITERIA 2: PROJECT SUCCESS MECHANISMS
The second aspect we assess is whether the project itself looks promising. When possible, we look at metrics of success that are specific to the project we’re assessing. When this isn’t available, we look at six mechanisms of success that any project needs to be successful.
Adoption -- Will people use the product or service?
Knowledge and abilities -- Can the team gain the ability needed to produce the product?
Production -- How will the team produce the product or service? Can they scale production?
Distribution -- How will the team distribute the product? Can they cost-effectively scale distribution?
Funding -- How will they sustainably receive funding? How will they ensure that their product is financially sustainable?
Recruitment -- How will they get the people needed to work on the project? Will they attract the necessary talent?
CRITERIA 3: TEAM COMPOSITION
We assess teams by evaluating the general abilities of the entrepreneurs and then evaluating the abilities needed to perform the idea. We place greater emphasis on the team the less established the project is.
Our team assessment model has eight parts, discussed below.
Teams require a wide variety of skills to succeed and without certain key skills teams are likely to fail. We evaluate eight different roles that all successful teams need. Based on the details of the project we are evaluating, we decide the importance of each role for the project and then evaluate how skilled the team is at each role. Note that multiple roles can be occupied by a single individual.
Strategy -- the ability to decide which tasks to undertake and how to perform them.
Persuasion -- the ability to get others to devote resources to the project.
R&D -- the ability to produce a superior product or service.
Coordination and conflict resolution -- the ability to share information across the team and resolve conflict.
Delegation -- the ability to free up time by having others perform tasks for you.
In addition, we look for all core members of the team to have the following core attributes:
Tenacity -- the ability to persevere through adversity. All core team members should have this ability.
Hard work -- the ability to put a large number of hours into the project. All core team members should have this ability.
- Shared values -- we prefer to invest in entrepreneurs who aim to create a flourishing world rather than those who aim for other incentives like money or prestige.
Our team composition model encompasses many pieces of conventional wisdom about startup team construction. Single-founder teams are penalized by the model because it is unlikely that all of the roles will occur in a single individual. Previously successful entrepreneurs have an advantage because entrepreneurial success is strong evidence of possessing the skills necessary for many of the roles. Conventional measures of career success like intelligence and conscientiousness are incorporated because they amplify effectiveness at all of the roles.