The Boundary Element Method (BEM)-based Finite Element Method (FEM) is an approach to approximate solutions of boundary-value problems over polygonal and polyhedral meshes. The approximation spaces are defined implicitly over the polygonal or polyhedral elements. Harmonic coordinates are recovered for the lowest order approximation in the special case of the diffusion problem. Thus, this method can be viewed as a generalization of harmonic coordinates to higher-order approximations. The definition and treatment of basis functions as well as the finite element formulation are discussed in detail. Furthermore, the BEM-based FEM is applied in the context of an adaptive FEM strategy yielding locally refined polygonal meshes. Theoretical evidence and numerical experiments are presented that establish optimal rates of convergence for uniform and adaptive mesh refinement strategies.