Fig. 3

vmrseq outperforms other methods in unsupervised clustering analysis of mouse frontal cortex data. a vmrseq’s UMAP visualization of neuron subpopulations annotated in the mouse frontal cortex dataset. Coordinates were computed from regional average methylation levels of VMRs. Broad cell classes are indicated by dashed outlines; subtypes are indicated by colored points and labeled by text. b Evaluation of clustering performance in terms of nearest neighbor count score. The x-axis is the number of CpG sites in varying numbers of top-ranked VMRs (log-scaled). We use number of CpGs instead of number of regions as x-axis because the size of detected regions varies significantly across methods (Additional File 1: Fig. S5d). Regions were ranked by metrics proposed in each method respectively (“Methods”). The dot size indicates number of included regions. The top 300, 1000, 3000, 10000, and 30000 regions (if applicable) and all selected regions were extracted from each method respectively for computing the score. vmrseq CRs do not have rank thus only represented by one dot. Line and point types distinguish granularity of the cell type labels. The score, ranging from 0 to 1, quantitatively evaluates the quality of clustering by averaging the proportions of neighbors that share the same label (see the “Methods” section for details). c Heatmap of regional average methylation level of top-ranked 500 VMRs from vmrseq. Rows are sorted by hierarchical clustering; dashed red squares are examples of potential cell-type-specific marker regions; white color indicates missing. d Marker regions for cell type mL4 detected by vmrseq exhibits more disparity of regional methylation level between the target and background cell types, compared to alternative methods. Annotated cell type labels from Luo et al. [12] were used to determine marker regions. Specifically, marker regions of cell type mL4 among detected regions of each method are defined as those with absolute difference \(> 0.2\) between the average methylation level of targeted cell type and all other cell types. Distribution of regional average methylation are plotted in violin shapes against cell subtypes; points represent methylation levels in individual cells