Organic photodetectors have promising applications in low-cost imaging, health monitoring and near infrared sensing. Recent research on organic photodetectors based on donor-acceptor systems has resulted in narrow-band, flexible and biocompatible devices, of which the best reach external photovoltaic quantum efficiencies approaching 100%. However, the high noise spectral density of these devices limits their specific detectivity to around 10^13 Jones in the visible and several orders of magnitude lower in the near-infrared, severely reducing performance. Here, we show that the shot noise, proportional to the dark current, dominates the noise spectral density, demanding a comprehensive understanding of the dark current. We demonstrate that, in addition to the intrinsic saturation current generated via charge-transfer states, dark current contains a major contribution from trap-assisted generated charges and decreases systematically with decreasing concentration of traps. By modeling the dark current of several donor-acceptor systems, we reveal the interplay between traps and charge-transfer states as source of dark current and show that traps dominate the generation processes, thus being the main limiting factor of organic photodetectors detectivity.