Is there any way to prevent users from launching and using remote desktop and to restrict it to only local admins or domain admins?

The reason being is that we do not want users to remote desktop home, but at the same time we want it to be available to certain users like administrators or power users.

Ideally there is a group policy that can be set to groups or users who have access to the remote desktop application from their machine.

Clarifications: I need the machine to be able to still have remote desktop work, just only with a specific user or group. The point is that we allow certain users to use remote desktop and others to not have access to it. There are machines where there are multiple users, so we cant just block a whole machine or by IP.

This needs to be done per a user account or login.