Background: While focusing on the management and care of COVID-19 patients, the mental health of these patients and their relatives is being overlooked. The aim of the current study was to measure anxiety and depression, and to assess their association with socio-demographic and pandemic-related stress factors in COVID-19 patients and their relatives during the initial stage of hospitalization. Methods: We assessed isolated hospitalized patients (N = 90) and their relatives (adults and children, N = 125) by phone, 25–72 h following patients' admission. The quantitative measures included the Anxiety and Depression modules of the Patient-Reported Outcomes Measurement Information System (PROMIS) and pandemic-related stress factors. Qualitative measures included questions exploring worries, sadness, and coping modes. Results: Both patients and relatives suffer from high levels of anxiety and related pandemic worries, with lower levels of depressive symptoms. Compared to adult relatives, child relatives reported significantly lower anxiety. The multivariable logistic regression analysis revealed an increased risk for anxiety among females and a decreased risk among ultra-orthodox participants. While increased anxiety among patients was associated with feelings of isolation, increased anxiety among relatives was associated with a feeling of not being protected by the hospital. Conclusions: Patients and relatives experience similar high anxiety levels which are more robust in women and lower in ultra-orthodox participants. Our findings indicate that anxiety symptoms of both patients and adult relatives should be addressed.