I think a small diminishing return on radius bonus (say -10% per module)
If you want a -X% sensor range per component system, you may as well impose a flat limit on the number of sensor components you can add to the ship and forget about the stacking penalty. The way that the stacking mechanics work, if you have a base sensor range of k, a percentile sensor bonus of c from sources other than sensor components, and a sensor component which provides a flat bonus of b and a percentile bonus of s to sensor range, the resulting sensor range for a vessel with n such components is
R(n) = (k + b * n) * (1 + c + s * n)
which for s < 0 has a maximum at
n = -(k * s + b + b * c) / (2 * b * s)
For b >> k, s = -0.1, and c = 0, that means that there is no reason to add more than 5 sensor components, as the fifth sensor component is the last sensor component that can possibly increase the ship's sensor range; components beyond that begin to reduce the vessel's sensor range, though the rounding of the computed range to an integer can conceal that for a few components (that rounding, however, can also mean that additional components cease to be useful before reaching the computed maximum useful number of components; for example, with b = 1, k = 2, c = 0, and s = -0.1, there is no point to adding more than 2 sensor components as the second sensor component brings the vessel to its maximum effective sensor range of 4).
I would also expect that there would be issues with communicating to the player what kind of benefit to expect from a technology which provides +Y% to sensor range. Using the parenthetical example and changing c from 0 to 0.1, the effective sensor range of a vessel with 2 sensor components does not change, and adding a third sensor to the vessel also does not improve the effective sensor range, which might lead the player to believe that the tech was worthless. A fourth sensor component, however, would allow the ship to attain its new maximum effective sensor range of 5, though the minimum sensor suite needed to achieve this costs twice what the minimum sensor suite needed for the maximum effective sensor range prior to the development of the tech needed, which leads one to question whether that +10% sensor range tech was worth much of anything.