How can the selection of innovation projects be structured to reduce false positives (investments that should not have been made) and false negatives (investments that should have been made but weren’t)? Prior research has provided important theoretical insights into organizational design determinants of selection regime performance, largely treating the set of alternatives available for selection as given. While this assumption has been relaxed in recent studies of how selection regimes affect idea generation and selection within an organization, ideas are increasingly generated outside organizational boundaries by innovators who can take their ideas to multiple organizations for evaluation. Our understanding of selection errors in real organizations facing these issues has been limited due to difficulties in collecting decision and outcome data on a complete set of proposed projects. We combine qualitative and quantitative data on a mobile application accelerator to understand how it implemented three different selection regimes over time. Using data on all 3,580 submissions to the accelerator, we collected the outcomes for all funded and rejected projects to measure false positives and negatives before empirically evaluating the three selection regimes’ effectiveness in avoiding both selection errors. Our findings suggest that in the last regime designed to increase the quality of submissions through additional layers of screening, evaluators were more likely to make false positives and negative decisions if one considers differences in the pools of projects submitted for selection. Additional analyses suggest adverse selection and over-weighting of applicant track records as likely mechanisms.