The Problem

Looping in custom transformers is a relatively old feature in FME - it dates back to 2006. However, blocking transformers (watch a short video about the difference between blocking and non-blocking transformers (See attachment: BlockingTransformers.mp4), such as the Clipper or SpatialRelator or any transformer that can use "Group By" functionality, could not work properly within loops.


It does not mean that they could not be included in a looping transformer, however they prevent any feature from going out via Transformer Loop port. Here is a short and rather poetic description of the problem by Graeme Hibert, one of our senior developers:

Conceptually, the feature processing loop of FME deals with one feature at a time. FME's greatest desire is to take one feature from the reader, push it through a string of transformers, and then feed the result to a writer before moving on to the next input feature. Transformers placed on the Workbench canvas can divert features along different paths, but in the end FME will push the single feature as far down the pipeline as possible before moving on to the next feature.

Workbench gives the user developing a custom transformer a tool which can thwart FME's natural process of transformation: the _transformer loop_ object allows them to take a feature from one point of processing within
the custom transformer and insert it into another point.  The devious trickery taking place under the covers leave
FME none the wiser: it happily continues the processing as if the feature had appeared in its new spot by natural means, continuing to push it as far down the pipeline as it can. 

There is a price to pay for this subterfuge, though.  Some of Workbench's transformers can _block_: that is,
they might hold on to some of the features that come into them, very often until the end of all of FME processing.  (This will happen when a transformer's "Group By" functionality is used, for instance, as no group can be known to be complete until every feature has been considered.) When a transformer holds onto a feature, FME goes back to the reader to get the next feature for processing. The blocking transformer cannot do its work -- and thus generate any output -- until all
of its input has been accumulated. 

This logically makes a blocking transformer ineligible for participation in a transformer loop, as the loop would occur at a time when the transformer's underlying factories are in a place where they are flushing their contents, and are not prepared to take on new features. 

The upshot is that there is no way to use a standard transformer loop when one of the transformers between the "loop in" point and the "loop out" point might block the normal flow of features. Users requiring iterative processing involving blocking transformers have had to take things to extraordinary measures, generally involving external processes -- Python scripts, separate applications, WorkspaceRunners, and so on -- in order to overcome this particular limitation. There has simply been no way for a workspace to invoke the necessary iteration within the main FME process.

 

Now this problem is in the past. It took a few iterations of really great development efforts as well as testing, but luckily, there were no blocking members in our team.

Despite the fact that the changes behind this functionality are big and not easy at all, we tried to make them as simple and unobtrusive for end user as possible.

There is only one limitation in using blocking transformers in loops - custom transformers containing them should be external (saved as FMX file) and always LINKED.

When a blocking transformer is placed in a custom trasnformer with a loop, or a LOOP port is added to a custom trasnformer with blocking transformers, FME warns a user with the following dialog:



If the user continues with the export, and creates an external custom transformer, blocking transformers will work within the loop as any other transformer.

Simple Example

Now let's have a look at a couple of examples that demonstrate this functionality.


We will begin with a realy simple one (See attachment: SimpleLoopDemo.zip). We will create a rectangle and a point within it, then, in a loop, we will move the point towards rectangle's boundary and check whether it stays within the rectangle. Once the point leaves the rectangle, the workspace should finish. Here is the workspace and the custom transformer with blocking SpatialFilter:



And the result of its work looks as follows:



There are a few things that help controlling the workflow within trasnformers with loops - a port called INCOMPLETE on the custom transformer, which is not exposed in the transformer's body, and in the parameters dialog - two mandatory parameters:



"Maximum number of iterations" allows avoiding infinite loops. FME will not prevent users from creating potentially infinite loops, when the translation will never end, however, it gives a chance to exit looping after a certain number of iterations. This means that setting this parameter to 0 can be dangerous if the logic of the transformer does not track potentially infinite loops.


"Attribute to hold interation count" creates an attribute that holds the number of times a feature passed through the loop.


The INCOMPLETE port outputs features that didn't meet the criteria that would allow them to get out via normal user defined port before the maximum number of iterations was reached.

Advanced Example

Now let's try a more realistic example (See attachment: LoopGeneralizer.zip) that uses all the parameters and the port mentioned above and does something useful.


Imagine we are working on a generalization project. We have two layers containing hydrography represented by lakes, and hypsography (contours). Our goal is to generalize contours without introducing new errors such as intersections with lakes. According to the project specification, the generalization tolerance for contours is 10 meters, however, if features are simplified excessively and make intersections with lakes, the tolerance can be decreased to 9, 8, or even 7 meters, but when 7 is not enough, we would like to inspect those contours visually and make a decision on a feature-by-feature basis. Features should be generalized with the highest tolerance possible, which means if a feature meets the condition wih the current tolerance, it is released from the loop via GENERALIZED port.


The workspace stays as simple as before:



The custom transformer now looks different - it takes care of getting published parameters, generalization, tracking tolerance changes etc.:



It gives the following result (gray represents original cotnours, brown shows generalized, and pink - untouched contours):



Note, that the contours are labelled with the tolerances they were generalized with, not elevations. Three outer contours worked with default project tolerance. The next three contours were generalized with tolerances 9, 8, and 7. All these contours left the transformer via GENERALIZED port. Finally, the last contour was not generalized at all. This was achieved by setting the maximum number of iteration to 4, that is, for tolerances 10, 9, 8, and 7. Features that weren't generalized (the most inner contour in this case) stayed untouched for our review and were output via INCOMPLETE port.


The workspace contains more details about the workflow.

Summary

• Starting with FME 2011, blocking transformers can participate in custom transformer loops;
• Custom transformers with blocking transformers within loops must be external FMX files and always linked to workspaces;
• The number of iterations can be controlled by "Maximum number of iterations" (recommended).