We study in detail sub-GeV dark matter scattering off electrons in xenon, including the expected electron recoil spectra and annual modulation spectra. We derive improved constraints using low-energy XENON10 and XENON100 ionization-only data. For XENON10, in addition to including electron-recoil data corresponding to about 1-3 electrons, we include for the first time events corresponding to about 4-7 electrons. Assuming the scattering is momentum independent (FDM=1), this strengthens a previous cross-section bound by almost an order of magnitude for dark matter masses above 50 MeV. The available XENON100 data corresponds to events with about 4-50 electrons, and leads to a constraint that is comparable to the XENON10 bound above 50 MeV for FDM=1. We demonstrate that a search for an annual modulation signal in upcoming xenon experiments (XENON1T, XENONnT, LZ) could substantially improve the above bounds even in the presence of large backgrounds. We also emphasize that in simple benchmark models of sub-GeV dark matter, the dark matter-electron scattering rate can be as high as one event every ten (two) seconds in the XENON1T (XENONnT or LZ) experiments, without being in conflict with any other known experimental bounds. While there are several sources of backgrounds that can produce single- or few-electron events, a large event rate can be consistent with a dark matter signal and should not be simply written off as purely a detector curiosity. This fact motivates a detailed analysis of the ionization-data ("S2") data, taking into account the expected annual modulation spectrum of the signal rate, as well as the DM-induced electron-recoil spectra, which are another powerful discriminant between signal and background.