From the People page, open the group you would like to remove the Vanta Mobile Monitor installation task from

Select the Device Monitoring option

Uncheck the Ask personnel to install Vanta Device Monitor option

To pass Vanta's tests, employee computers must be equipped with a mobile device management (MDM) tool or the Vanta Device Monitor. Make sure you have connected an MDM or configured the Vanta Device Monitor.

The changes you make to a Task set apply to every group with the Task Set assigned to them.

If you do not want these changes to apply to all users but only to a specific subset, you will need to create a new group for that subset of users and then create and assign a new Task Set to that group with the desired 'Require device monitoring settings'.