An exactly soluble model, in which a random exchange ± J and uniform field B can be transformed into a uniform exchange and a random magnetic field, ± B, is studied. At B = 0, the model exhibits a spin glass ordered phase, for which all critical exponents are derived. In particular, the magnetic susceptibility exhibits a cusp and its divergent second derivative is related to long range correlations of a spin glass order parameter. At B ≠ 0, the thermodynamic singularities disappear for isotropic (Ising) systems at dimensions d < 4(2), and are distinct from those at B = 0 for 6 > d > 4(2). A more general scaling theory is formulated for B → 0.