The terrestrial subsurface is one of the least explored environments on Earth. Due to the difficult accessibility of rocks and sediments below ground, most microbial inventories have focused solely on groundwater. Therefore, endolithic, and sessile microbes that live on/in the subsurface sediments and rocks have received little interest. This study aims to (1) investigate these communities by linking their diversity and abundance to abiotic factors; and (2) to study compositional differences in the microbial communities detected in groundwater compared to those detected in the surrounding solid materials. We extracted microbial DNA from two aquifer systems in the northern Laurentians (Quebec, Canada). Planktonic microbial communities were sampled in porous sand and sand-clay aquifers. During well construction, samples of rocks and sediments were recovered from the exposed sediment horizon down to the bedrock (depth 42 m and 50 m) at 1.5 m depth intervals. To analyze microbial community diversity, archaeal and bacterial 16S rRNA genes were sequenced (Illumina miseq). Results show that depth and pH influence the diversity and the amount of identified microbial taxa. Furthermore, our analysis reveals that subsurface microbiota are dominated by Proteobacteria, Acidobacteria and Actinobacteriota. Despite this similarity, evidence was found of a unique rock matrix microbiome compared to shallower sediments, and to the groundwater. These findings suggest these environmental factors are effectives habitat filters, restricting community membership to progressively more narrowly defined lineages as they vary. The existence of similar taxa in both rock matrix and surrounding groundwater indicates that interactions and exchange between planktonic groundwater microorganisms and those that are attached to aquifer sediments and rocks may affect the assembly of groundwater microbial communities. It also proves the importance of sampling both liquid and solid phases of an aquifer.