From the People page, open the group you would like to remove Agent installation task from

Select the Device Monitoring option

Uncheck the Ask personnel to install Vanta Agent option

To pass Vanta's tests, employee computers must install a mobile device management (MDM) tool or the Vanta Agent. Make sure you have connected an MDM or configured the Vanta Agent.

The changes you make to a Task set apply to every group with the Task Sets assigned to them.

If you do not want these changes to apply to all users but a specific subset, you would need to create a new group for that subset of users and then create and assign a new Task Set to that group with the desired 'Require device monitoring settings'.