Linked server available to users without permissions

I have users seeing a linked server which they are not supposed to see.

The linked server is defined so that only I will have access to it, but everybody can see and use it.

I have created the new linked server using the following steps:

1. Connect SSMS Object Explorer to the SQL Server instance
2. Expand Server Objects, right click Linked Servers and left-click New Linked Server...
3. Under General tab choose SQL Server for Server type and write the Server’s name
4. Under Security tab in the upper part click Add, choose “sa” as Local Login, enter Remote User‘s name and Remote Password
5. Under Security tab in the lower part (under For a login not defined in the list above, connections will:) choose the first option: Not be made
6. Click OK and start testing

Now the only person supposed to see the linked server is me (“sa”), but somehow other users can see it and use it.

Note 1: The users who can use the linked server have permissions on the remote server, they are not seeing data that they should not see, they are just able to access it from the linked server when they should not be able to.

Note 2: I am the only sysadmin on both instances.