Interplay of Spatial and Topological Defects in Polymer Networks