Interested in FME & Python? Checkout the recording of the June 2012 Python webinar on safe.com
IntroductionPython is a programming language that can be used within FME to accomplish tasks either before or after FME runs or to perform tasks within FME which are not possible with standard FME tools and transformers*. For example you can use a python script to extract source data from a zip file before FME runs or to send an email after FME runs. In this article we will look at simple examples of python scripting in the 4 main places python is used in FME:
For some basic resources on programming in python see the FMEPedia article: What is Python and How Can I Install It? http://fmepedia.safe.com/articles/FAQ/What-is-Python-and-How-Can-I-Install-It
Python and FME Basics
FME installs its own python interpreter which is used by any scripts running python in FME. If you want FME to use your own python interpreter there is a setting in Workbench. Please see this article for more information:http://fmepedia.safe.com/articles/How_To/Choosing-a-different-Python-Interpreter-installation
FME Objects and PythonMuch of FME's core functionality is available within python scripts by using FME classes and methods from the FME Objects python API. For example you can get at the FME log file, FME parameters and use any of the countless FME methods such as getArea or reproject etc. (more on this later). To use FME Objects in your python script with FME 2012 and newer you will need to import FME Objects with the statement:
For more information on the new FME Objects python API please see the article here:
Python-related transformers have excellent Help right in FME Workbench, which also includes Help for Startup and Shutdown scripts. For using FME Objects in Python, you can find complete documentation of the Python FME Objects API here: <FME>\fmeobjects\python\apidoc\index.html
FME Variables in Python Scripts
A number of FME related variables are available in FME Startup and Shutdown scripts. As of FME 2014 these variables are accessible in the fme module so you will need to include the statement import fme to access them. For example the fme.status variable can be true or false at the end of a translation depending on the success or failure of the workspace. For a complete list of FME variables available in Startup and Shutdown Python scripts please go to Help > Workspace Basics > User Interface Reference > Workspace Navigator > Workspace Parameters > Advanced > Startup and Shutdown Python Scripts.
FME Parameters in Python Scripts
One of the FME related variables described above returns a dictionary of FME parameters and their values when the workspace was run. These parameter values are often useful in a python script, for example when you want to know file path of the source dataset used in a reader or some other user defined parameter. In FME 2014 you can get the values of FME parameters by accessing the dictionary fme.macroValues. For example if you want to get the value of the source dataset parameter for your AutoCAD reader and the parameter is called SourceDataset_ACAD and you can access this parameter in python like this:
import fme SourceDataset = fme.macroValues['SourceDataset_ACAD']
Please note: if using an FME version older than FME 2014 use the syntax below:
SourceDataset = FME_MacroValues['SourceDataset_ACAD']
Startup Python ScriptIn the Workbench Navigator pane under Workbench Parameters - > Advanced you will see a parameter called Startup Python Script. Here you can enter some Python code that will execute before the FME translation.
Note: If you want to set or return an FME parameter this is not place - see Scripted Parameters below.
Example of Startup Python ScriptAttached workspace: StartupPython.fmw
By default when FME writes to an existing Microsoft Excel file we insert new records, but what if you want to delete the Excel file each time? (See the article http://fmepedia.safe.com/articles/How_To/Example-scripts-for-deleting-Excel-files-prior-to-writing)
We can delete the destination dataset before running FME using the following Start Up Python script (workspace attached):
import os OutputFile = str(FME_MacroValues['DestDataset_XLS_ADO'])FME World Tour 2013 - 1Spatial & Star-Apic (London and Leeds) print OutputFile if os.path.exists(OutputFile): os.remove(OutputFile)
We are getting the value for the destination dataset parameter fomr hte FME_MacroValues dictionary. Your destination dataset parameter may have another name and which you can find by right clicking on the parameter and selecting Edit User Parameter Definition where you can see the Name:
Scripted parameters are extremely useful when we want to set a parameter in FME based on something we derived or calculated from another parameter or parameters. For example you may want users to select themes or groups of layers and have your script set the the individual feature types to read within these groups.
PythonCaller TransformerThe PythonCaller transformer lets you do things to features or groups of features using a Python script. Before using this transformer please be sure there is not already an FME transformer which does the task you want to do. Feel free to ask us in support (www.safe.com/support) if there is a transformer that can help you.
The PythonCaller can call a Function which you can use to process one feature at a time or a Class where you may want to do things to groups of features. In either case the EntryPoint parameter of the transformer is where you name the function or class in the script to run. The PythonCaller can use the full range of FME Objects including numerous methods and classes.
PythonCaller Example 1 (Function)
Attached workspace: PythonCallerFunction.fmw
import fmeobjects import time # Template Function interface: def timestampFeature(feature): curTime = time.ctime(time.time()) feature.setAttribute("timestamp", curTime)The script uses some FME objects methods so we need import fmeobjects and we are using the python time module so we need to import it as well. The function definition accepts FME feature as its only argument meaning all features will enter the function one by one for processing.
A new attribute is added to the feature with setAttribute method on the feature - this is actually an FME Objects method.
3. Click OK to dismiss the code editor. In the transformer you need set the Entry Point to the name of the function timestampFeature. Because we have added a new attribute called timestamp we can expose it by entering its name in Attributes to Expose.
4. Use a creator to create some features to send you the PythonCaller and a Logger to look at the output. Every feature should have a timestamp attribute.
PythonCaller Example 2 (Function)The FME Store includes a custom transformer called the FuzzyStringComparer which uses the Python difflib module to compare two string attributes and calculate a similarity ratio. In some ways this is a better example because it is something we cannot do with a regular FME transformer.
1. Open a blank workspace and add the FME Store transformer FuzzyStringComparer by typing it on the canvas or by browsing the FME Store transformers in the Tranformer gallery. Right-click on the transformer and select Embed which will allow you to edit the transformer and see its contents in the same workspace.
2. Once the FuzzyStringComparer is embedded right click on it to Edit and see the contents. Locate the PythonCaller transformer and edit its properties to access the script editor. Notice again we are using a function - this time to compare string attributes on a feature by feature basis. Again we are setting a new attribute which is called FuzzyStringCompare.ratio. An AttributeExposer is used later in the workflow to access the attribute we have created.
string1 = feature.getStringAttribute('FuzzyStringCompare.string1').lower()
string2 = feature.getStringAttribute('FuzzyStringCompare.string2').lower()
ratio = difflib.SequenceMatcher(None,string1,string2).ratio()
PythonCaller Example 3 (Class)Attached workpsace: PythonCallerClass.fmw
Again we can use the sample provided in the Transformer description for the PythonCaller. This time we will use a class rather than a function to calculate the area of all features. Once again this is not an ideal example because this task could be done with standard FME transformers.
1. Start a new blank workspace and use a Creator to create a polygon feature. Add a PythonCaller transformer and a logger to see the output.
2. Open the PythonCaller and open the script in the code editor. Copy in the code from the example in transformer description or from below. You may need to add some line returns and indentation to the code so that looks like this (remember indents matter in python):
import fmeobjects class MyFeatureProcessor(object):
def __init__(self): self.featureList =  self.totalArea = 0.0 def input(self,feature): self.featureList.append(feature) self.totalArea += feature.getGeometry().getArea() def close(self): for feature in self.featureList: feature.setAttribute("total_area", self.totalArea) self.pyoutput(feature)
Again we are using a number of FME Objects methods so we need to import fmeobjects. On the input method we add each feature to a list and get the area from each feature which is added to the totalArea. On the close method we loop through each feature and add the total_area attribute to each one. It is important to note that if we want features to continue through the workspace they must be written out using the pyoutput() method.
3. Click OK to dismiss the code editor. In the transformer you need set the Entry Point to the name of the Class MyFeatureProcessor. Because we have added a new attribute called total_area we can expose it by entering its name in Attributes to Expose.
4. Run the workspace to ensure you have the total_area attribute on your feature. Open the Creator transformer and increase the number of features. Run the worspace again to see that the total area of all features is being calculated.
Shutdown Python ScriptA python script can be added to a workspace which will run when the workspace completes. For example you may want to copy files somewhere when a workspace is done, zip data, or send an email. The script runs after all of the transformers reader and writers in a workspace are finished but the script is still run a part of the FME process. You can access the FME parameter values in the same way as in the examples above using the FME_MacroValues dictionary. A shutdown python script can also access a number of global variables which FME sets such as FME_FeaturesRead, FME_FeaturesWritten and many more. For a complete list of the global variables please see the FME Fundamentals Help under the topic FME_END_PYTHON.
Example of Shutdown Python ScriptAttached workspace: ShutdownPython.fmw
In this example we add a shutdown python script to a workspace which sends an email using gmail when the workspace finishes. You can edit the script to use other mail servers is you wish. The subject of the email will say whether the workspace was successful or not and include the number of features written. A similar example can also be found in FME Workbench -> Help -> Startup and Shutdown Python Scripts.
1. Open any workspace and in the Workbench Navigator pane under Workbench parameters - >Advanced you will see a parameter called Shutdown Python Script. Here you can enter the Python code below which will can send the email using a gmail account. The comments in the code below signified with # explain most of the script. Notice two key points:
a) We are getting the user parameters from the FME_MacroValues dictionary in the same way as in the other scripts above. For example the gmail user name can be set when the workspace is run and comes from the published parameter like this: FME_MacroValues['GmailUser'].
b) Secondly, notice we use two global variables from FME: FME_FeaturesWritten which is a python dictionary, and FME_Status which is a boolean and tells us if the workspace succeed or failed.
# The smtplib gives us the emailing tools import smtplib, fmeobjects # Get the FME parameter values set when workspace run to = FME_MacroValues['EmailTo'] gmail_user = FME_MacroValues['GmailUser'] gmail_pwd = FME_MacroValues['Password'] Subject = FME_MacroValues['SuccessSubject'] FailSubject = FME_MacroValues['FailSubject'] Emailfrom = FME_MacroValues['EmailFrom'] #Get number of features FME wrote FeaturesWritten = str(FME_FeaturesWritten) # If FME Fails then Change Subject # FME_Status tells if FME was successful status = FME_Status if status == 0: Subject = FailSubject Message = 'Workspace Failed' else: Message = 'Workspace Successful the following features were written: ' + FeaturesWritten # Sending the email # Using the gmail smtp server #You can edit this line to use your own email server smtpserver = smtplib.SMTP("smtp.gmail.com",587) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo smtpserver.login(gmail_user, gmail_pwd) header = 'To:' + to + '\n' + 'From: ' + Emailfrom + '\n' + 'Subject:' + Subject +'\n' print header msg = header + '\n' + Message + '\n\n' smtpserver.sendmail(gmail_user, to, msg) print 'done!' smtpserver.close()
2. Run the workspace using File - Prompt and Run or the prompt and run button. Fill in the appropriate parameters. You will need your own gmail account user name (email address) and password and a from email address which should be the same as the user. You can also fill in the to email address and a subject line for failure and success. In the attached sample workspace the Fail parameter is used to simulate workspace failure by send the features to the Terminator transformer.
For a more complex shutdown python examples see the articles:
Notify an FME Server Topic on Success or Failure of a Workspace
Example Workflow using FME, Python and Oracle