We investigate twisted double bilayer graphene (TDBG), a four-layer system composed of two AB-stacked graphene bilayers rotated with respect to each other by a small angle. Our ab-initio band structure calculations reveal a considerable energy gap at the charge point neutrality that we assign to the intrinsic symmetric polarization (ISP). We then introduce the ISP effect into the tight-binding parameterization and perform calculations on TDBG models that include lattice relaxation effects down to very small twist angles. We identify a narrow region around the magic angle θ*= 1.3° characterized by a manifold of remarkably flat bands gapped out from other states even without external electric fields. To understand the fundamental origin of the magic angle in TDBG, we construct a continuum model that points to a hidden mathematical link to the twisted bilayer graphene (TBG) model, thus indicating that the band flattening is a fundamental feature of TDBG, and is not a result of external fields.