As part of filtering irrelevant information from entering visual working memory (VWM) and selecting only the relevant information for further processing the system should first tag the pieces of information as relevant or irrelevant. We manipulated difficulty of tagging items as relevant or irrelevant by applying perceptual grouping cues to investigate if it can improve filtering performance in VWM. Participants performed a change-detection task with three targets, six targets, or three targets and three distractors (filtering condition) in the memory display, and were asked to remember the colors (Experiments 1–2) or the orientations (Experiments 3–5) of the targets and ignore the distractors. In the filtering conditions, either the targets (Experiments 1, 3, and 4) or the distractors (Experiments 2 and 5) formed an illusory object (a Kanizsa triangle), appeared in a triangle-like configuration (grouping by proximity), or appeared at random positions (non-grouping). Grouping the targets improved filtering performance relative to non-grouping. Moreover, the illusory object cue further improved filtering performance beyond a proximity cue, but only when the cue was compatible with the task. When the distractors were grouped, the proximity cue improved filtering performance, and the illusory object cue, despite being a potent grouping cue, failed to improve filtering performance when it was compatible with the task. We suggest that the grouping cues advanced tagging of the grouped items. Yet, when the grouping cue strongly enhanced processing of the distractors, the tagging failed, such that the preliminary process of estimating incoming items led to full processing of the grouped items.