There are several things here that can help you with your current situation.
First, you can add a Prefix/Suffix to your FileSet on a GET as in:
This attaches the value (like a Date/TimeStamp) to the Beginning or End of each file being retrieved. Also, utilizing the Variables during the Get:
This allows you to directly address the files on the Destination (Local Server) or the Source (Remote Server). Thus, allowing you to specifically go back and delete only the files you downloaded from the Remote Server with just a single Remote File List Variable associated to the "Processed Source Files Variable".
The "Destination Files Variable" is a Local File List Variable that will allow you to perform actions on that list as well, once the files have been retrieved to your local system.
One other thing that might help ... GoAnywhere Director has a function on the Advanced (s)FTP(s) tabs that allow you to utilize Persistent Connections.
This means you make the single FTP connection to GET the files from the Source Server. By using this Persistent Variable ID later in your project to delete the files on the Source Server, you don't have to make another connection, the connection remains open from the original GET of the files.
Hope this helps!