We address the problem of passive estimation of the Time-Difference of Arrival (TDOA) of an unknown, stochastic signal, at two sensors. The key question addressed here is whether additional sensors, receiving the same signal at various relative delays, can serve to improve the accuracy in estimating the TDOA of interest between the first two sensors (without exploiting any underlying parameterization, such as dependence on the transmitter's location). We derive the Cramér-Rao Lower Bound (CRLB) on the resulting joint estimation error in a model which possibly includes multipath reflections. We show analytically, that in a multipath-free scenario, at high to moderate Signal to Noise Ratios, additional sensors do not offer any improvement in accuracy. However, we also demonstrate (numerically) that in the presence of multipath reflections (possibly received at all sensors), the additional sensors can indeed assist in estimating the TDOA of interest with improved accuracy.