Interested in FME & Python? Checkout the recording of the June 2012 Python webinar on safe.com

Introduction

Python is a programming language that can be used within FME to accomplish tasks either before or after FME runs or to perform tasks within FME which are not possible with standard FME tools and transformers*.  For example you can use a python script to extract source data from a zip file before FME runs or to send an email after FME runs.  In this article we will look at simple examples of python scripting in the 4 main places python is used in FME:
  • Startup Python Script
  • Scripted Parameter
  • PythonCaller and PythonCreator Transformers
  • Shutdown Python Script
*Important Note: There are numerous FME transformers for performing almost every task you can think of without resorting to scripting.  If you are thinking about writing a python script to do something in FME you should make absolutely sure there is not an existing transformer available already. Please feel free to ask us by contacting support at www.safe.com/support.

For some basic resources on programming in python see the FMEPedia article: What is Python and How Can I Install It? http://fmepedia.safe.com/articles/FAQ/What-is-Python-and-How-Can-I-Install-It

Python and FME Basics

Python Interpreter

FME installs its own python interpreter which is used by any scripts running python in FME. If you want FME to use your own python interpreter there is a setting in Workbench. Please see this article for more information:

http://fmepedia.safe.com/articles/How_To/Choosing-a-different-Python-Interpreter-installation

FME Objects and Python

Much of FME's core functionality is available within python scripts by using FME classes and methods from the FME Objects python API.  For example you can get at the FME log file, FME parameters and use any of the countless FME methods such as getArea or reproject etc. (more on this later). To use FME Objects in your python script with FME 2012 and newer you will need to import FME Objects with the statement: 
import fmeobjects

For more information on the new FME Objects python API please see the article here:
http://fmepedia.safe.com/articles/How_To/New-Python-FME-Objects-API-in-FME-2012

Documentation

Python-related transformers have excellent Help right in FME Workbench, which also includes Help for Startup and Shutdown scripts. For using FME Objects in Python, you can find complete documentation of the Python FME Objects API here: <FME>\fmeobjects\python\apidoc\index.html

FME Variables in Python Scripts

A number of FME related variables are available in FME Startup and Shutdown scripts. As of FME 2014 these variables are accessible in the fme module so you will need to include the statement import fme to access them. For example the fme.status variable can be true or false at the end of a translation depending on the success or failure of the workspace.  For a complete list of FME variables available in Startup and Shutdown Python scripts please go to Help > Workspace Basics > User Interface Reference > Workspace Navigator > Workspace Parameters > Advanced > Startup and Shutdown Python Scripts.

Prior to FME 2014 these variables were named differently and it was not necessary to import the fme module.  For example the status was returned in the variable FME_Status. For these older versions of FME the FME variables are documented in the FME Quick Translator Help.

FME Parameters in Python Scripts

One of the FME related variables described above returns a dictionary of FME parameters and their values when the workspace was run.  These parameter values are often useful in a python script, for example when you want to know file path of the source dataset used in a reader or some other user defined parameter. In FME 2014 you can get the values of FME parameters by accessing the dictionary fme.macroValues.  For example if you want to get the value of the source dataset parameter for your AutoCAD reader and the parameter is called SourceDataset_ACAD and you can access this parameter in python like this:
 

import fme 
SourceDataset = fme.macroValues['SourceDataset_ACAD']

Please note: if using an FME version older than FME 2014 use the syntax below:

SourceDataset = FME_MacroValues['SourceDataset_ACAD']


We can also set FME parameters by creating a scripted parameter and returning the value from the script. This is covered below in the section on Scripted Parameters.
 

Startup Python Script

In the Workbench Navigator pane under Workbench Parameters - > Advanced you will see a parameter called Startup Python Script. Here you can enter some Python code that will execute before the FME translation.

User-added image

Note: If you want to set or return an FME parameter this is not place - see Scripted Parameters below.

Example of Startup Python Script 

Attached workspace: StartupPython.fmw

By default when FME writes to an existing Microsoft Excel file we insert new records, but what if you want to delete the Excel file each time? (See the article http://fmepedia.safe.com/articles/How_To/Example-scripts-for-deleting-Excel-files-prior-to-writing)


We can delete the destination dataset before running FME using the following Start Up Python script (workspace attached):
import os 
OutputFile = str(FME_MacroValues['DestDataset_XLS_ADO'])FME World Tour 2013 - 1Spatial & Star-Apic (London and Leeds)
print OutputFile
if os.path.exists(OutputFile):
   os.remove(OutputFile)

We are getting the value for the destination dataset parameter fomr hte FME_MacroValues[] dictionary.  Your destination dataset parameter may have another name and which you can find by right clicking on the parameter and selecting Edit User Parameter Definition where you can see the Name:

User-added image


Scripted Parameters

Scripted parameters are extremely useful when we want to set a parameter in FME based on something we derived or calculated from another parameter or parameters. For example you may want users to select themes or groups of layers and have your script set the the individual feature types to read within these groups.

Example Scripted Parameter

Attached workspace: ScriptedParameter.fmw or see FME Store

The FME Store has a good example of scripting the feature types to read parameter based on a group selected by the user. In FME Workbench you can access this workspace from the FME Store by using the menu File - New and browsing the Store on the bottom left. There is a group of workspaces called Scripting in WB - select the one called Python scripted parameters - layer groups and click Choose to download and open.


Use the File - Prompt and Run to see the published parameters. Notice that the layers you are promoted for are groups: Bus Information, All Transit, Road and Rail. Choose Bus Information and run the workspace. Notice that the workspace read both the BusRoutes and the BusStops feature types.

Lets look at the scripted parameter used to tell FME that choosing Bus Information should return both of these feature types to the Feature Types to Read parameter.

In the navigator pane expand User Parameters -> Private Parameters and you will see a python scripted parameter called feature_types_to_read. Notice that this parameter is linked to the actual Feature Types to Read parameter of the reader in the workspace. Whatever value the script returns will be used by the SDF reader to determine which feature type to read.

User-added image


Go back to the scripted parameter and double click on it and click the ellipsis to open the editor.  Here is the script used inside:
featureTypes = ''
if FME_MacroValues['layers'].find('Bus Information') != -1:
    featureTypes += 'BusRoutes BusStops '
if FME_MacroValues['layers'].find('Road and Rail') != -1:
    featureTypes += 'metrorail Roads '
if FME_MacroValues['layers'].find('All Transit') != -1:
    featureTypes += 'metrorail Roads BusRoutes BusStops Labels '
#Debug
#print featureTypes
return featureTypes

You can see that we have a series of if statements which find out which layer the user chose using the FME_MacroValues[ ] dictionary and then sets the value of featureTypes which will be returned. The last line is the most important as this is where we "return" the actual value of the scripted paramater with statement
return featureTypes
This return statement must always exist in a python scripted parameter because it is here that the parameter value is given to FME.  Also notice the commented out print statement.  You can use print to help you debug your scripts by returning variable values to the FME log. For more in logging with python see this article: http://fmepedia.safe.com/articles/Error_Unexpected_Behavior/Logging-with-Python-scripts

Example Scripted Parameter 2 (Complex)

Attached workspace:UnZipSource.fmw
If you understand the basic scripted parameter example above you should have no trouble understanding this more complex example. The attached workspace UnZipSource.fmw accepts a published parameter for a zip file and a parameter for a directory to unzip to.  A scripted parameter unzips the file and returns the directory to the source dataset parameter of the reader.


PythonCaller Transformer

The PythonCaller transformer lets you do things to features or groups of features using a Python script. Before using this transformer please be sure there is not already an FME transformer which does the task you want to do. Feel free to ask us in support (www.safe.com/support) if there is a transformer that can help you.

The PythonCaller can call a Function which you can use to process one feature at a time or a Class where you may want to do things to groups of features. In either case the EntryPoint parameter of the transformer is where you name the function or class in the script to run. The PythonCaller can use the full range of FME Objects including numerous methods and classes.

PythonCaller Example 1 (Function)

Attached workspace: PythonCallerFunction.fmw
As with all transformers FME Workbench will display a detailed transformer description when you have added the transformer to the canvas and have it selected. In this case we will use the sample code included in the description to add a timestamp to features. This is not an ideal example because this task could be done with the standard FME transformer the TimeStamper.

1. Add a PythonCaller to your workspace and select it. You should see the Transformer Description in a window below. If you don't see it there maybe a tab for it beside the log window or you can open it from the menu - View -> Window -> Transformer Description.  

2. Open the transformer properties and click on the ellipsis to open the code editor. Notice there is already a Template  Function and Class there for you. Delete the class because we are just using a function for now. Rename the function timestamper and copy in the code from example in the transformer description or from below.  You may need to add some line returns and indentation to the code so that it looks like this (remember indents matter in python):

import fmeobjects
import time 
# Template Function interface:

def timestampFeature(feature):
    curTime = time.ctime(time.time())
    feature.setAttribute("timestamp", curTime)
The script uses some FME objects methods so we need import fmeobjects and we are using the python time module so we need to import it as well. The function definition accepts FME feature as its only argument meaning all features will enter the function one by one for processing. 

A new attribute is added to the feature with setAttribute method on the feature - this is actually an FME Objects method.  


3. Click OK to dismiss the code editor.  In the transformer you need set the Entry Point to the name of the function timestampFeature.  Because we have added a new attribute called timestamp we can expose it by entering its name in Attributes to Expose. 

User-added image

4. Use a creator to create some features to send you the PythonCaller and a Logger to look at the output. Every feature should have a timestamp attribute.

PythonCaller Example 2 (Function)

The FME Store includes a custom transformer called the FuzzyStringComparer which uses the Python difflib module to compare two string attributes and calculate a similarity ratio. In some ways this is a better example because it is something we cannot do with a regular FME transformer.

1. Open a blank workspace and add the FME Store transformer FuzzyStringComparer by typing it on the canvas or by browsing the FME Store transformers in the Tranformer gallery.  Right-click on the transformer and select Embed which will allow you to edit the transformer and see its contents in the same workspace.

Embedding the custom transformer


2. Once the FuzzyStringComparer is embedded right click on it to Edit and see the contents. Locate the PythonCaller transformer and edit its properties to access the script editor. Notice again we are using a function - this time to compare string attributes on a feature by feature basis. Again we are setting a new attribute which is called FuzzyStringCompare.ratio. An AttributeExposer is used later in the workflow to access the attribute we have created.

import difflib

def FuzzyStringCompare(feature):

    string1 = feature.getStringAttribute('FuzzyStringCompare.string1').lower()

    string2 = feature.getStringAttribute('FuzzyStringCompare.string2').lower()

    ratio = difflib.SequenceMatcher(None,string1,string2).ratio()

    feature.setRealAttribute('FuzzyStringCompare.ratio',ratio)

Important Note:
In the above example the function used to get an FME attribute value is from the now deprecated pyfme.  If you are using fmeobjects the function to use would be feature.getAttribute()
 

PythonCaller Example 3 (Class)

Attached workpsace: PythonCallerClass.fmw
Again we can use the sample provided in the Transformer description for the PythonCaller. This time we will use a class rather than a function to calculate the area of all features. Once again this is not an ideal example because this task could be done with standard FME transformers.

1. Start a new blank workspace and use a Creator to create a polygon feature.  Add a PythonCaller transformer and a logger to see the output.

2. Open the PythonCaller and open the script in the code editor.  Copy in the code from the example in transformer description or from below. You may need to add some line returns and indentation to the code so that looks like this (remember indents matter in python):
 
import fmeobjects
class MyFeatureProcessor(object):
    def __init__(self):
        self.featureList = []
        self.totalArea = 0.0

    def input(self,feature):
        self.featureList.append(feature)
        self.totalArea += feature.getGeometry().getArea()

      def close(self):
         for feature in self.featureList:
                feature.setAttribute("total_area", self.totalArea)
                self.pyoutput(feature)

Again we are using a number of FME Objects methods so we need to import fmeobjects. On the input method we add each feature to a list and get the area from each feature which is added to the totalArea.  On the close method we loop through each feature and add the total_area attribute to each one.  It is important to note that if we want features to continue through the workspace they must be written out using the pyoutput() method.

3. Click OK to dismiss the code editor.  In the transformer you need set the Entry Point to the name of the Class MyFeatureProcessor.  Because we have added a new attribute called total_area we can expose it by entering its name in Attributes to Expose. 
 
User-added image

4. Run the workspace to ensure you have the total_area attribute on your feature. Open the Creator transformer and increase the number of features. Run the worspace again to see that the total area of all features is being calculated.

Shutdown Python Script

A python script can be added to a workspace which will run when the workspace completes.  For example you may want to copy files somewhere when a workspace is done, zip data, or send an email.  The script runs after all of the transformers reader and writers in a workspace are finished but the script is still run a part of the FME process.  You can access the FME parameter values in the same way as in the examples above using the  FME_MacroValues[] dictionary. A shutdown python script can also access a number of global variables which FME sets such as FME_FeaturesRead, FME_FeaturesWritten and many more. For a complete list of the global variables please see the FME Fundamentals Help under the topic FME_END_PYTHON.

Example of Shutdown Python Script 

Attached workspace: ShutdownPython.fmw

In this example we add a shutdown python script to a workspace which sends an email using gmail when the workspace finishes. You can edit the script to use other mail servers is you wish. The subject of the email will say whether the workspace was successful or not and include the number of features written. A similar example can also be found in FME Workbench -> Help -> Startup and Shutdown Python Scripts.

1. Open any workspace and in the Workbench Navigator pane under Workbench parameters - >Advanced you will see a parameter called Shutdown Python Script. Here you can enter the Python code below which will can send the email using a gmail account.  The comments in the code below signified with # explain most of the script. Notice two key points:
a) We are getting the user parameters from the FME_MacroValues dictionary in the same way as in the other scripts above. For example the gmail user name can be set when the workspace is run and  comes from the published parameter like this:  FME_MacroValues['GmailUser'].  
b) Secondly, notice we use two global variables from FME: FME_FeaturesWritten which is a python dictionary, and FME_Status which is a boolean and tells us if the workspace succeed or failed.
 
# The smtplib gives us the emailing tools

import smtplib, fmeobjects

# Get the FME parameter values set when workspace run

to =  FME_MacroValues['EmailTo']
gmail_user =  FME_MacroValues['GmailUser']
gmail_pwd =  FME_MacroValues['Password']
Subject =  FME_MacroValues['SuccessSubject']
FailSubject =  FME_MacroValues['FailSubject']
Emailfrom =  FME_MacroValues['EmailFrom']

#Get number of features FME wrote
FeaturesWritten = str(FME_FeaturesWritten)

# If FME Fails then Change Subject
# FME_Status tells if FME was successful
status = FME_Status
if status == 0:
    Subject =  FailSubject
    Message = 'Workspace Failed'    
else:
    Message = 'Workspace Successful the following features were written: ' + FeaturesWritten
# Sending the email
# Using the gmail smtp server
#You can edit this line to use your own email server
smtpserver = smtplib.SMTP("smtp.gmail.com",587)
smtpserver.ehlo()
smtpserver.starttls()
smtpserver.ehlo
smtpserver.login(gmail_user, gmail_pwd)
header = 'To:' + to + '\n' + 'From: ' + Emailfrom + '\n' + 'Subject:' + Subject +'\n'
print header
msg = header + '\n' + Message + '\n\n'
smtpserver.sendmail(gmail_user, to, msg)
print 'done!'
smtpserver.close()

2. Run the workspace using File - Prompt and Run or the prompt and run button. Fill in the appropriate parameters. You will need your own gmail account user name (email address) and password and a from email address which should be the same as the user. You can also fill in the to email address and a subject line for failure and success.  In the attached sample workspace the Fail parameter is used to simulate workspace failure by send the features to the Terminator transformer.

User-added image

For a more complex shutdown python examples see the articles:
Notify an FME Server Topic on Success or Failure of a Workspace
(http://fmepedia.safe.com/articles/How_To/Notify-an-FME-Server-Topic-on-Success-or-Failure-of-a-Workspace)
Example Workflow using FME, Python and Oracle
(http://fmepedia.safe.com/articles/Samples_and_Demos/Example-Workflow-using-FME-Python-and-Oracle)