One of the key elements of the boundary reform that went through between the 2019 and 2023 elections was to ensure that constituencies have broadly equal numbers of electors. Prior to this there had been more variation (and a few big anomalies), whereas the boundary reform means that all seats now have an electorate of 73,393 +/-5%. (I think this was a pretty uncontroversial change but had been held up for years because the Tories kept trying to accompany it with a change to the number of MPs, which was a lot more controversial.)

I haven’t bothered doing the calculation on a seat-by-seat basis, but the electorate distribution would have to skew really badly in favour of the +5%s being Lab/Con and the -5%s being non-Lab/Con for your concern to come to fruition.