Working with Folder Lists

Looking for an example project to get you started?
1 post Page 1 of 1

Support_Rick

Support Specialist
Posts: 592
Joined: Tue Jul 17, 2012 2:12 pm
Location: Phoenix, AZ

Post by Support_Rick » Tue Sep 23, 2014 5:25 pm
Working with Folders

GoAnywhere Director provides numerous tasks for file management, such as rename, copy, move and delete. GoAnywhere tasks also provide you with the ability to create and filter a list of files to process as a group, or one file at a time. What about folders, is it possible to manage folders as well?

The answer is, yes!

The process requires the help of built in commands from your operating system. Instead of providing you with a tip this month, we are providing a trick that will help you identify and manage folder lists. This will be accomplished using a Project that relies on the Execute Command task to create a directory listing. The directory list will identify specific folders based on a naming pattern, and the project can be called at any time from any other Project. (Think of this Project as if it is a utility, it can be created once and then used over and over again in your instance of GoAnywhere Director.)

The ‘Get FolderList‘ project below demonstrates how the utility Project is called from the primary Project. To call this utility Project, you must pass three variables as parameters.
  1. Root Folder: The Folder to use as the base search location
  2. Pattern: Wildcard pattern of folder name
  3. Recursive: Y or N : Continue search to sub-folders (Default = N)

The utility project will take the values passed and execute a local command to the Operating System on which GoAnywhere Director is installed. The final result of this call to the utility project is a flat file being created in the WorkSpace with a list of folders identified by the pattern value. All that is necessary at that point is to read the flat file and generate a FolderList RowSet.

There are a few requirements and assumptions based on installation Operating System:

*NOTE* Project is written for GoAnywhere Director Version 4.6 and above.

All OS’s:
  • A WorkSpace has been created in the Primary Project prior to calling the Get FolderList Utility project
  • Folder patterns must match OS folder naming conventions
  • Permissions to execute local commands has been granted to GoAnywhere Director and/or the ResourceID

IBMi:
  • Assumes that a ResourceID has been created and it matches the System Variable (system.systemName) for executing commands locally on your IBMi. Change this value to match your ResourceID, if necessary.
The ‘Get FolderList’ utility project can be called from any project where a generated folder list is needed.

To keep this simple, I usually create a “Utility” folder within GoAnywhere Director and place utility projects like this inside that folder. I can then call this project from any business project (Production, Test or Development) when I need to identify a folder list for exercising a business need.

As an example (for Windows OS), let’s assume we need to identify specific folders from a local server that contain today’s date.

Our folder structure for this test would look something like this (Image 1), located in the "C:\Temp\FolderList" root.
Image 1.png
Image 1.png (7.77 KiB) Viewed 6874 times
The Primary Project would look something like the following. The format of the folders for Windows includes the "\".
Code: Select all
<project name="Test Windows FolderList" mainModule="Main" version="2.0" loglevel="silent">

	<module name="Main">

		<createWorkspace version="1.0" />

		<callProject label="Get FolderList Windows" project="/Utilities/Get Folder List" runInSameJob="true" inheritUserVariables="true" returnUserVariables="true" mode="interactive" version="1.0">
			<variable name="pRootFolder" value="C:\Temp\FolderList" />
			<variable name="pPattern"    value="NFS*" />
			<variable name="pRecursive"  value="Y" />
		</callProject>

		<readFlatFile label="Read FolderList" inputFile="${FList}" outputRowSetVariable="FolderList" trim="both" version="1.0" />

		<forEachLoop label="Loop through FolderList" itemsVariable="${FolderList}" currentItemVariable="ThisFolder">

			<print label="(Status) Print Dir Name" version="1.0">
				<![CDATA[
==============================
${ThisFolder[1]}
==============================
        ]]>
			</print>

		</forEachLoop>

		<deleteWorkspace version="1.0" />
	</module>

</project>
Now, let’s look at the utility Project named “Get FolderList Windows” that will retrieve the desired folder list. Depending on the OS version selected and using operating system commands, the utility will generate the folder list and place it into the WorkSpace in a file called FolderList.

*Windows Version*
Code: Select all
<project name="Get FolderList Windows" mainModule="Main" version="2.0" logLevel="silent" threadSafe="true" >

  <variable name="pRootFolder"  value="" />
  <variable name="pPattern"     value="" />
  <variable name="pRecursive"   value="N" />

  <module name="Main">

    <if label="Parameter Check" condition="${ isEmpty( pRootFolder ) or isEmpty( pPattern ) }">
      <print label="(Status) Parameter Check" version="1.0">
        <![CDATA[
================================================================================
**Parameter Check** 
Parameter values cannot be blank:
${system.lineFeed}
Root Location:   ${ pRootFolder }
Folder Pattern:  ${ pPattern }
Recursive?       ${ pRecursive } 
================================================================================
        ]]>
      </print>
      <exitModule/>
    </if>
    
    <print label="(Status) Get Started" version="1.0">
      <![CDATA[
================================================================================
**STATUS** Getting Folders for ${system.os.name} OS...

Root Location:   ${pRootFolder}
Folder Pattern:  ${pPattern}
Recursive?       ${ If( pRecursive == 'Y', 'Yes', 'No' ) } 
================================================================================
      ]]>
    </print>

    <callModule label="(Module) Get Folders" module="Get Folders" version="1.0" />

  </module>

  <module name="Get Folders">

    <setVariable label="SetVar: FList"   name="FList" value="${ Concat( String( system.job.workspace ), '\FolderList' ) }"  version="2.0" />

    <setVariable label="SetVar: Subs"   name="Subs" value="${ if( Contains( pRecursive, 'y', false ), '/S', system.emptyString ) }"  version="2.0" />

    <print label="Create Command File" file="${system.job.workspace}\CrtDirList.bat" append="false" version="1.0">
	<![CDATA[
@Echo Off                                            ${system.carriageReturn}${system.lineFeed}
CD /D ${pRootFolder}                                 ${system.carriageReturn}${system.lineFeed}
Dir ${pPattern} /ad /b ${Subs} > "${FList}"          ${system.carriageReturn}${system.lineFeed}
	]]>
    </print>

    <exec label="Generate File List" executable="${system.job.workspace}\CrtDirList.bat" workingDir="${system.job.workspace}" version="1.0" />

  </module>

  <description>
Generates a file called "FOLDERLIST" in the current WorkSpace.  
Allows you to read FOLDERLIST as a FlatFile for generating a Rowset to process desired Folder List.  

** Assumptions **
1. GoAnywhere Director version 4.6 and above
2. Create WorkSpace has been executed on Parent Project
  </description>

</project>
Now that the pieces are in place, we can execute the primary Project to demonstrate how it calls the utility Project and makes the folder selection.

The primary Project passes the following parameter variables to the utility Project:
  • RootFolder = “C:\Temp\FolderList”
  • Pattern = “${ CurrentDate(‘yyyyMM’)}*” -- This creates the pattern “201408*”
  • Recursive = “Y”
The Project JobLog contains the following:
Code: Select all
Job Number: 1373030453173 
Project Name: /Tips and Tricks/Test Windows FolderList
Submitted By: RElliott
System: GAD_01
GoAnywhere Director 4.6.2 running on Windows 7 6.1 (amd64)
Overriding variable 'pRootFolder' with value 'C:\Temp\FolderList'
Overriding variable 'pPattern' with value '201408*'
Overriding variable 'pRecursive' with value 'N'

================================================================================
**STATUS** Getting Folders for Windows OS...
Root Location:   C:\Temp\FolderList
Folder Pattern:  201408*
Recursive?       No 
================================================================================
==============================
C:\temp\FolderList\20140823-Folder A
==============================
==============================
C:\temp\FolderList\20140824-Folder A
==============================
==============================
C:\temp\FolderList\20140825-Folder A
==============================
Changing the Primary Project to pass the following parameter variables:
  • RootFolder = “C:\Temp\FolderList”
  • Pattern = “*${CurrentDate(‘2014*dd’)}*” -- This creates the pattern “*2014*25*”
  • Recursive = “N”
Produces the following:
Code: Select all
================================================================================
**STATUS** Getting Folders for Windows OS...
Root Location:   C:\Temp\FolderList
Folder Pattern:  *2014*25*
Recursive?       No 
================================================================================
==============================
C:\temp\FolderList\20140725-Folder A
==============================
==============================
C:\temp\FolderList\20140825-Folder A
==============================
As you can see, the Get FolderList Utility Project provides you with a way to generate a FolderList RowSet variable to process a pattern across any/all folders within a local folder structure that will make the filtered list available to your GoAnywhere Project.

*Please Note*
This utility is presented as is and is only an example of the functionality of the GoAnywhere Product. We strongly suggest you fully test and validate before using in a Production Environment.


Additional OS Versions Below
*===============================================================================*


*Linux/Unix Version*
Code: Select all
<project name="Get FolderList Linux" mainModule="Main" version="2.0" logLevel="silent" threadSafe="true" >

  <variable name="pRootFolder"  value="" />
  <variable name="pPattern"     value="" />
  <variable name="pRecursive"   value="N" />

  <module name="Main">

    <if label="Parameter Check" condition="${ isEmpty( pRootFolder ) or isEmpty( pPattern ) }">
      <print label="(Status) Parameter Check" version="1.0">
        <![CDATA[
================================================================================
**Parameter Check**
Parameter values cannot be blank:
${system.lineFeed}
Root Location:   ${ pRootFolder }
Folder Pattern:  ${ pPattern }
Recursive?       ${ pRecursive }
================================================================================
        ]]>
      </print>
      <exitModule/>
    </if>
   
    <setVariable label="Set: DblQuote" name="DblQuote" value=""" version="2.0" />

    <print label="(Status) Get Started" version="1.0">
      <![CDATA[
================================================================================
**STATUS** Getting Folders for ${system.os.name} OS...

Root Location:   ${pRootFolder}
Folder Pattern:  ${pPattern}
Recursive?       ${ If( pRecursive == 'Y', 'Yes', 'No' ) }
================================================================================
      ]]>
    </print>

    <callModule label="(Module) Get Folders" module="Get Folders" version="1.0" />

  </module>

  <module name="Get Folders">

		<print label="(Status) Get Started" version="1.0">
			<![CDATA[
================================================================================
**STATUS** Linux/Unix Module
================================================================================
      ]]>
		</print>

		<setVariable label="SetVar: FList"   name="FList" value="${ Concat( String( system.job.workspace ), '/FolderList' ) }"  version="2.0" />
		<setVariable label="SetVar: Cmd" name="Cmd" value="${ if( Contains( pRecursive, &apos;y&apos;, false ),          
    Concat( &apos;find &apos;, DblQuote, pRootFolder, DblQuote, &apos; -type d -name &apos;, DblQuote, pPattern, DblQuote, &apos; | sort -f | sed &apos;, DblQuote, &apos;s|^\./||g&apos;, DblQuote, &apos; > &apos;, ' FolderList' ),          
    Concat( &apos;ls -d &apos;, pRootFolder,&apos;/&apos;,pPattern, &apos; | sort -f | sed &apos;, DblQuote, &apos;s|^\./||g&apos;, DblQuote, &apos; > &apos;, ' FolderList' ) ) }" version="2.0" />

		<print label="Create Command File" file="${system.job.workspace}/CrtDirList.sh" append="false" version="1.0">
			<![CDATA[
${Cmd}
      ]]>
		</print>

		<exec label="Make Command Executable" executable="chmod" workingDir="${system.job.workspace}" version="1.0">
			<arg value="+x" />
			<arg value="${system.job.workspace}/CrtDirList.sh" />
		</exec>

		<exec label="Generate File List" executable="${system.job.workspace}/CrtDirList.sh" workingDir="${system.job.workspace}" version="1.0" />

</module>

<description>
Generates a file called "FOLDERLIST" in the current WorkSpace. 
Allows you to read FOLDERLIST as a FlatFile for generating a Rowset to process desired Folder List. 

** Assumptions **
1. GoAnywhere Director version 4.6 and above
2. Create WorkSpace has been executed on Parent Project
</description>

</project>
*IBMi Version*
Code: Select all
<project name="Get FolderList IBMi" mainModule="Main" version="2.0" logLevel="silent" threadSafe="true" >

  <variable name="pRootFolder"  value="" />
  <variable name="pPattern"     value="" />
  <variable name="pRecursive"   value="N" />

  <module name="Main">

    <if label="Parameter Check" condition="${ isEmpty( pRootFolder ) or isEmpty( pPattern ) }">
      <print label="(Status) Parameter Check" version="1.0">
        <![CDATA[
================================================================================
**Parameter Check**
Parameter values cannot be blank:
${system.lineFeed}
Root Location:   ${ pRootFolder }
Folder Pattern:  ${ pPattern }
Recursive?       ${ pRecursive }
================================================================================
        ]]>
      </print>
      <exitModule/>
    </if>
   
    <setVariable label="Set: DblQuote" name="DblQuote" value=""" version="2.0" />

    <print label="(Status) Get Started" version="1.0">
      <![CDATA[
================================================================================
**STATUS** Getting Folders for ${system.os.name} OS...

Root Location:   ${pRootFolder}
Folder Pattern:  ${pPattern}
Recursive?       ${ If( pRecursive == 'Y', 'Yes', 'No' ) }
================================================================================
      ]]>
    </print>

    <callModule label="(Module) Get Folders" module="Get Folders" version="1.0" />

  </module>

  <module name="Get Folders">
  
		<print label="(Status) Get Started" version="1.0">
			<![CDATA[
================================================================================
**STATUS** IBMi Module
================================================================================
      ]]>
		</print>

		<setVariable label="SetVar: FList" name="FList" value="${ Concat( String( system.job.workspace ), &apos;/FolderList&apos; ) }" version="2.0" />

		<setVariable label="SetVar: Touch" name="Touch" value="${ Concat( 'touch -C 819 ', FList ) }" version="2.0" />
		<setVariable label="SetVar: Cmd" name="Cmd" value="${ if( Contains( pRecursive, &apos;y&apos;, false ),               
    Concat( &apos;find &apos;, DblQuote, pRootFolder, DblQuote, &apos; -type d -name &apos;, DblQuote, pPattern, DblQuote, &apos; | sort -f | sed &apos;, DblQuote, &apos;s|^\./||g&apos;, DblQuote, &apos; >> &apos;, FList ),               
    Concat( &apos;ls -d &apos;, pRootFolder,&apos;/&apos;,pPattern, &apos; | sort -f | sed &apos;, DblQuote, &apos;s|^\./||g&apos;, DblQuote, &apos; >> &apos;, FList ) ) }" version="2.0" />

		<exec400 label="Generate File List" resourceId="${system.systemName}" version="1.0">
			<command threadSafe="true">
				<![CDATA[qsh cmd('${Touch}')]]>
			</command>
			<command threadSafe="true">
				<![CDATA[qsh cmd('${Cmd}')]]>
			</command>
		</exec400>

  </module>

<description>
Generates a file called "FOLDERLIST" in the current WorkSpace. 
Allows you to read FOLDERLIST as a FlatFile for generating a Rowset to process desired Folder List. 

** Assumptions **
1. GoAnywhere Director version 4.6 and above
2. Create WorkSpace has been executed on Parent Project
</description>

</project>
Feel free to contact Linoma Software with any questions or to inquire about our On-Site Professional Services or Training Programs.
Rick Elliott
Lead Solutions Consultant
(402) 944.4242
(800) 949-4696
1 post Page 1 of 1