Preambles

Preambles are a template of comments and directives that are used when mapping files are generated. They are used to define comments and directives that are included in mapping files generated for source/destination, source, and destination metafiles (respectively).


There are also two more preambles for workbench-only things. (See the SHAPE metafile for an example.) Directives within the workbench preamble sections are only inserted when writing a workspace. The reason we have a separate section is so that you can only put the essential GUI prompts in the regular metafile sections, so that when people do a drag and drop translation it will only prompt for essential things, but you can expose ALL the format's parameters in workbench, which has a nice tree view for parameter editing.


There are four preamble sections in total:

  • SOURCE_PREAMBLE: Preamble for readers, goes into both Universal Translator and Workbench translations.
  • WORKBENCH_SOURCE_PREAMBLE: Preamble for readers, used in Workbench translations, commented out in mapping files.
  • DESTINATION_PREAMBLE: Preamble for writers, goes into both Universal Translator and Workbench translations.
  • WORKBENCH_DESTINATION_PREAMBLE: Preamble for writers, used in Workbench translations, commented out in mapping files.


The reason we have two sets of preambles is because some options may not make sense in a Quick Translator translation. These options go into the WORKBENCH_ preamble, so users can customize the translation further in Workbench. For a good example of this, see the SHAPE format metafile (<InstallFolder>\metafiles\shape.fmf), which has all four sections.


One more note on this: You cannot assume that your preamble will be all alone in a workspace. There may be two copies of it (say, you have two AUTOCAD readers), so any items that need to be unique (say, a factory name) need to be prefixed. You can use the WB_KEYWORD macro to prepend to factory names or anything else of interest. See the AUTOCAD source preamble metafile (autocadSourcePreamble.fmi) for an example of this.

Metafile Directives (Universal Translator & Workbench)

SOURCE_READER

Syntax: SOURCE_READER <READER NAME> [<PARM NAME> <PARM_VALUE>] [[-]<Schema Keyword> <Schema Macro|Constant Value>]

This directive specifies the name of the reader used to drive the generation of a mapping file/workspace. It also is where parameters specific to how the reader is used are specified.

Values include:

SCHEMA_MODE LAYERS_ONLY

  • This is only valid for DWG/DXF and causes a mapping file to be generated which only uses the layer names from the input file. Schema Scan directives are:
    • IGNORE_ATTRIBUTES_WITH_PREFIX
      • This directive identifies all of the attributes that are not to be used for the purposes of generating a schema. There is no limit to the number of the ignore statements which may be specified. For example, in AutoCAD there are a large number of attributes that begin with extended_data. If we want to ignore these then we simply define the following:
      • e.g. IGNORE_ATTRIBUTES_WITH_PREFIX extended_data
  • KEEP_ATTRIBUTE
    • This directive is used to identify attributes that are to be added to the reader schema in all cases, when performing a FEATURE_SCAN. For example, in AutoCAD the value is autocad_elevation. This indicates that no matter if the autocad_elevation attribute is found in the feature scan, the attribute is added to the schema.
  • GEOMETRY_TYPE_FIELD
    • This directive is used to identify the attribute that specifies the different geometry types. For example, in AutoCAD the value is autocad_entity. This is the value that controls the attribution specific to a particular type of geometry.
  • GEOMETRY_ATTRIBUTE_PREFIX
    • This specifies the prefix for the attributes which are to be associated with the geometry. For example, in autocad the attributes that begin with autocad_ identify the geometry attributes.

SCHEMA_MODE FEATURE_SCAN

  • This value can be used for any reader and results in the source reader reading all features. In this mode the features are scanned and a schema is constructed. This is useful for formats that do not explicitly store schema information. When this schema mode is specified then the reader’s readSchema method is not used.

SCHEMA_MODE FILE_BASED_SCHEMA SCHEMA_FILE_NAME <basename>.sch

  • This indicates the reader's schemas are canned/fixed/static, and that they are defined in FME_HOME/schemas/<basename>.sch. As a result, IFMEReader::readSchema() does not return any schema features. This is the preferred method of implementing canned schemas, in opposition to achieving the same result programmatically using IFMEReader::readSchema().

SCAN_MAX_FEATURES <number>

  • This causes the schema scanner to quit after receiving <number> features, useful for large datasets when the entire schema is known after a first few features.

Schema Keywords

Lastly, the SOURCE_READER line specifies keywords to be used during schema generation. The following conventions should be followed

  • Use the hyphen.
    • This means the keyword will now be accessible from the mapping file, allowing us to share code used when reading features instead of schemas
    • NOTE: In non-FME Objects situations, the values will not persist in the mapping file object between readSchema() and read()
  • The schema keyword should be identical to the mapping file keyword but with the format type prefix (i.e. MYFORMAT_READ_RAW_DATA)
  • The macro name should be identical to the schema keyword, but without the format type prefix (i.e. READ_RAW_DATA)

Here's an example:

 SOURCE_READER MYFORMAT  -MYFORMAT_READ_RAW_DATA           $(READ_RAW_DATA) \
                         -MYFORMAT_READ_QUALITY_DATA       NO                     \
                         -MYFORMAT_ADD_QUALITY_REFERENCE   $(ADD_QUALITY_REFERENCE) \
                         -MYFORMAT_CNIG_NAME_COMPLETION    $(CNIG_NAME_COMPLETION)  \
                         -MYFORMAT_ERROR_OUTPUT            $(ERROR_OUTPUT)

FORMAT_NAME <FORMAT NAME>

This directive simply defines the name of the format for which the file describes the mapping file generation.

SOURCE_KEYWORD <KEYWORD>

This is an optional directive. If not specified then the <format name> is used as the keyword. This is rarely used. If this is not specified and the reader and writer are the same, the mapping file generation will generate a unique name of the form <format name>_IN for this entry.

DESTINATION_KEYWORD <KEYWORD>

This is an optional directive similar to SOURCE_KEYWORD. If this is not specified and the reader and writer are the same the mapping file generation will generate a unique name of the form <format name>_OUT for this entry.

SOURCE_DATASET <SOURCE DATASET>

This statement defines the value which is placed on the mapping file dataset line. Typical values for this will be SOURCE_DATASET $[SourceDataset]

READER_META_ATTRIBUTES

This directive allows readers to selectively add fme_basename, fme_dataset, and fme_feature_type attributes to all features they read.

For example, a reader might specify

$(FMEGEN_SOURCE_KEYWORD)_READER_META_ATTRIBUTES formatname

to add a formatname attribute to each feature.

DESTINATION_DATASET <DESTINATION DATASET>

This statement defines the value which is placed on the mapping file dataset line. Typical values for this will be DESTINATION_DATASET $[DestDataset]

SOURCE_SETTINGS/END_SOURCE_SETTINGS

Enables the creation of metafile-based settings boxes for the reader. The section must begin with SOURCE_SETTINGS, end with END_SOURCE_SETTINGS, and must be placed outside the SOURCE_PREAMBLE/END_SOURCE_PREAMBLE section.

The Settings Box Title will be placed in the title bar of the generated dialog. The default title will be generated as: <Input|Output> Settings for <FormatLongName>

To override this default setting, add this line directly after the SOURCE_SETTINGS line:

GUI TITLE <Title>

Next, specify the GUI elements. If you don't supply any GUI elements then no settings box will be created, even if you specify the GUI TITLE described above. For each setting, two lines must be specified:

[-]DEFAULT_VALUE <keyword> <default value>
[-]GUI [OPTIONAL] <guiType> <keyword> <keyword description>

where:

  • -DEFAULT_VALUE indicates that the line is used for setting box generation only. Otherwise, the GUI line will be written to the mapping file ad verbum.
  • <keyword> is the name of the keyword for which the setting will be used. It is also useful to know that this is the name of the macro for the keyword.
  • <default value> is the desired default value. To specify no default, use "".
  • [-] should only be specified if the keyword is to be set and used during schema generation, and then cannot be changed when running the mapping file/workspace.
  • [OPTIONAL] determines whether the field is optional. Omitting this token makes the field mandatory. The user must give all mandatory fields some value, otherwise the Ok button on the dialog will be disabled.
  • <guiType> is one of the possible types for GUI lines in metafiles. This includes TEXT, PASSWORD, DIRNAME, FILENAME, CHOICE, and ACTIVECHOICE. If you use CHOICE or ACTIVECHOICE, then after <keyword> will be another field (not described in the spec above): a '%' separated list of possible choices. See the GUI documentation in the FME foundation manual for more details on the syntax of the various GUI types.
  • <keyword description> is the label used for the setting.

The second line of the two line pair has the same syntax as other usage of GUI lines within the metafile.

Settings Box Best Practices

These best practices ensure a consistent feel between all the settings boxes.

  1. once you’ve begun using group boxes, you're committed to placing all widgets within the current settings box within some group box
  2. checkbox widgets must always be in groups, and at the top of the groups they’re in (this improves readability)
  3. avoid ending group box descriptions with a colon

Example 1: JSON

 SOURCE_SETTINGS  
!---------------------------------------------------------------------- 
! The title for the source settings dialog box.
!----------------------------------------------------------------------

GUI TITLE JSON Source Settings  
!---------------------------------------------------------------------- 
! Delete the download file? 
!----------------------------------------------------------------------
DEFAULT_VALUE DELETE_DOWNLOAD_FILE Yes
GUI CHOICE DELETE_DOWNLOAD_FILE Yes%No Delete downloaded file - [for URL datasets only]

!---------------------------------------------------------------------- 
! Get a proxy URL 
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_URL ""
GUI OPTIONAL TEXT PROXY_URL Http Proxy URL:  

!---------------------------------------------------------------------- 
! Get a proxy port number  
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_PORT ""
GUI OPTIONAL TEXT PROXY_PORT Http Proxy Port:  

!---------------------------------------------------------------------- 
! Get a proxy user name 
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_USERNAME ""
GUI OPTIONAL TEXT PROXY_USERNAME Http Proxy Username:  

!---------------------------------------------------------------------- 
! Get a proxy password 
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_PASSWORD ""
GUI OPTIONAL PASSWORD PROXY_PASSWORD Http Proxy Password:

!---------------------------------------------------------------------- 
! Get a proxy authentication type 
!----------------------------------------------------------------------
DEFAULT_VALUE PROXY_AUTH_METHOD Basic
GUI OPTIONAL CHOICE PROXY_AUTH_METHOD  Basic%Digest Http Proxy Authentication Method: 

END_SOURCE_SETTINGS

 

Example 2: Feature Type Picker

SOURCE_SETTINGS  GUI TITLE Microsoft Access Input Settings  
GUI GROUP DATASET%PASSWORD Database Connection  
GUI GROUP TABLELIST%WHERE_CLAUSE Constraints  
# ===========================================================================  
# The ADO provider in use.  For Access, this is always MDB_ADO  DEFAULT_VALUE PROVIDER_TYPE MDB_ADO  # ===========================================================================  # This will show a widget to select dataset on the setting box. The  # preceding "-" will ensure that DEFAULT_MACRO line and KEYWORD line is not  # written out to the mapping file.  -DEFAULT_VALUE DATASET  -GUI FILENAME_MUSTEXIST DATASET Access_Database_Files(*.mdb)|*.mdb|All_files(*.*)|*.* Database Path:  # ===========================================================================  # Optional Password  DEFAULT_VALUE PASSWORD  GUI OPTIONAL PASSWORD PASSWORD Password:  # ===========================================================================  # This will show a feature type selector widget on the dialog. In order for it  # to work all the connection string parameters must be defined within the scope  # of this setting box.  -DEFAULT_VALUE TABLELIST  -GUI OPTIONAL FEATURE_TYPES TABLELIST $[DATASET],PROVIDER_TYPE,$[PROVIDER_TYPE],PASSWORD,$[PASSWORD] Table List:  # ===========================================================================  # Optional global WHERE clause  DEFAULT_VALUE WHERE_CLAUSE  GUI OPTIONAL TEXT WHERE_CLAUSE Where Clause:  END_SOURCE_SETTINGS

 

Example 3: Add a Search Envelope

  # ===========================================================================  # To show search envelope settings this is how the GUI and DEFAULT_VALUE lines  # should like. Macro name MUST be like SEARCH_ENVELOPE_xxxx..., where "xxxx..." could  # could be used to make the macro names unique.  # ===========================================================================  -DEFAULT_VALUE USE_SEARCH_ENVELOPE NO  -GUI ACTIVEGROUP USE_SEARCH_ENVELOPE SEARCH_ENVELOPE_MINX%SEARCH_ENVELOPE_MINY%SEARCH_ENVELOPE_MAXX%SEARCH_ENVELOPE_MAXY%CLIP_TO_ENVELOPE Use Search Envelope  # this LOOKUP_GLOBAL has a drawback where the value of 0, if assigned to any parameters, will be treated as if no value has been set to that parameter and a  # different value has to be used in UT. It is recommended to use "LOOKUP [parameter name] <Unused>, [value]" for compatibility with UT  -GUI LOOKUP_GLOBAL <Unused>,0  -GUI LOOKUP CLIP_TO_ENVELOPE <Unused>,No  # ===========================================================================  DEFAULT_VALUE SEARCH_ENVELOPE_MINX 0  GUI FLOAT SEARCH_ENVELOPE_MINX Minimum X:  __  # ===========================================================================  DEFAULT_VALUE SEARCH_ENVELOPE_MINY 0  GUI FLOAT SEARCH_ENVELOPE_MINY Minimum Y:  __  # ===========================================================================  DEFAULT_VALUE SEARCH_ENVELOPE_MAXX 0  GUI FLOAT SEARCH_ENVELOPE_MAXX Maximum X:  __  # ===========================================================================  DEFAULT_VALUE SEARCH_ENVELOPE_MAXY 0  GUI FLOAT SEARCH_ENVELOPE_MAXY Maximum Y  __  # ===========================================================================  DEFAULT_VALUE CLIP_TO_ENVELOPE No  GUI CHECKBOX CLIP_TO_ENVELOPE Yes%No Clip to Search Envelope

DESTINATION_SETTINGS/END_DESTINATION_SETTINGS

This is the same as the SOURCE_SETTINGS/END_SOURCE_SETTINGS above except it is for the writer.

DEF_LINE_TEMPLATE

This will determine the syntax for the DEF lines that mapping files and workbench produce.

It is important that any parameters you define in the WORKBENCH_DEFLINE_PARMS section appear here, or else they won't actually show up in the DEF lines. If any parameters are defined on the DEF_LINE_TEMPLATE, Workbench will automatically add everything defined in WORKBENCH_DEFLINE_PARMS.

If {FME_GEN_GEOMETRY} is provided on a def line, then DEF_LINE_BREAK must have the value GEOM_CHANGE.

This provides a similar function in mapping files, with the exception that WORKBENCH_DEFLINE_PARMS is irrelevant since it is only for Workbench; however since your format should work in Workbench you must still use it.

DEF_LINE_TEMPLATE defines the format of the defline. It is of the form

DEF_LINE_TEMPLATE <groupName> [<def_attr> <def_attr_value>]*

where <def_attr> is the name of a definition attribute and <def_attr_value> is the value of the definition line attribute. The value for group name may be hardcoded or may be specified using one of two special keywords which are available during the definition phase of the correlation table generation.

  • {FME_GEN_GROUP_NAME} is the current name of the group as defined by the input feature. The value of this macro is driven totally by the source side of the format.
  • {FME_GEN_GROUP_NUMBER} is the current group number. This macro is also totally driven by the source side. This is useful when you want to ensure that you assign unique numbers to different definition lines.

<def_attr_value> must be carefully handled in order to work in all possible environments: FME Workbench, Generic Translations, Quick Translator Generate and Run. The rules are as follows:

  • Empty strings must be escape quote delimited: \"\"
  • Strings with no spaces may be optionally quote delimited: "Object_ID" or Object_ID
    • GENTRANS will treat escaped quotes around non-empty strings as significant.
  • Strings with spaces must be quote and escape quote delimited: "\"Object ID\""

Here are a few examples:

MapInfo

DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME} 
SDE

DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME}      \
                           SDE_BASE_LAYER 0 \
                           SDE_LAYER
$(FME_GEN_GROUP_NUMBER)                            \
                           SDE_GRID{0} 10000       \
                           SDE_DIMENSION 0         \
                           SDE_AVERAGE_PTS 75      \
                           SDE_INIT_NUM_FEATS 2000 \
                           SDE_GEOMETRY()          \
                           SDE_CONFIG_KEYWORD
DEFAULTS

 

ARCGEN

DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME} \
ARCGEN_GEOMETRY {FME_GEN_GEOMETRY}

Shape

DEF_LINE_TEMPLATE {FME_GEN_GROUP_NAME}
SHAPE_GEOMETRY {FME_GEN_GEOMETRY}

DEF_LINE_BREAK

This affects how DEF lines are created within Workbench/Universal Translator (UT).

If the line contains:

  • GEOM_CHANGE, then only single geometry per feature type will be allowed, and the 'Geometry' combo box in Workbench will be enabled in the feature type properties dialog.
  • ATTRIB_CHANGE, then each feature type can handle all geometry types, and 'Geometry' will be disabled in the Workbench combo box.
  • NO_DEF, no DEF lines will be written at all, and the geometry combo box in Workbench will be disabled.
  • NO_ATTRIBS, then no user-defined attributes will be written onto the DEF line.

Note that a single line can contain more than one of these values in it (i.e. for BC MOEP the line is: DEF_LINE_BREAK ATTRIB_CHANGE NO_ATTRIBS).

GEOM_MAP

This section defines how a format's types map into fme transfer types. For simplicity's sake, just think of transfer types as though they were fme types (i.e. fme_type). It is important that there are format types that map to each fme transfer type. The complete list of fme transfer types along with their permitted attributes is:

  • fme_point
    • fme_color
    • fme_symbol
    • fme_size
  • fme_line
    • fme_color
    • fme_style
    • fme_width
    • fme_smooth_line
  • fme_polygon
    • fme_fill_style
    • fme_foreground_color
    • fme_background_color
    • fme_color
    • fme_style
    • fme_width
  • fme_text
    • fme_rotation
    • fme_oblique_angle
    • fme_background_color
    • fme_color
    • fme_font
    • fme_text_style
    • fme_text_size
    • fme_justification
    • fme_text_string
    • fme_spacing
    • fme_text_width
  • fme_ellipse
    • fme_primary_axis
    • fme_secondary_axis
    • fme_rotation
    • fme_fill_style
    • fme_foreground_color
    • fme_background_color
    • fme_color
    • fme_style
    • fme_width
  • fme_arc
    • fme_primary_axis
    • fme_secondary_axis
    • fme_start_angle
    • fme_sweep_angle
    • fme_rotation
    • fme_color
    • fme_style
    • fme_width
    • fme_smooth_line
  • fme_rectangle
    • fme_fill_style
    • fme_foreground_color
    • fme_background_color
    • fme_color
    • fme_style
    • fme_width
  • fme_rounded_rectangle
    • fme_rounding
    • fme_fill_style
    • fme_foreground_color
    • fme_background_color
    • fme_color
    • fme_style
    • fme_width
  • fme_raster
  • fme_no_geom
  • fme_raster (implied mapping: fme_polygon)?
  • fme_collection (implied mapping: fme_no_geom)
  • fme_solid (implied mapping: fme_line)
  • fme_surface (implied mapping: fme_line)
  • fme_point_cloud (implied mapping: fme_polygon)

Some fme_type values have an "implied mapping". If no explicit mapping is found, the same mapping is used as for the implied type. For example, for mapinfo, fme_raster will be mapped to mapinfo_region (the mapinfo mapping for fme_polygon).

It is absolutely valid that multiple fme transfer types map to the same format type. For example, geodb_polygon maps to fme_polygon, fme_rounded_rectangle, and fme_rectangle. It is also possible for a single fme transfer type to map to multiple format types. Continuing the example with Geodatabase, geodb_polyline, geodb_simple_edge, and geodb_complex_edge all map to fme_line. In this situation however, the order of these mappings (i.e. which is first in the file) is very important because when writing to the format, fme_line will get mapped to the first match found in the metafile, which in the Geodatabase case would be geodb_polyline.

Within Workbench, this section lists all the possible geometry types this format supports. It is important that a mapping to all fme transfer types be present, for if Workbench cannot map a generic type (say, fme_line), bad things will happen.

Look at existing formats to see examples for this section. For good reading on this that describes the bare basics needed to know, check out: FME Universal Translator help -> FME Fundamentals -> FMEFundamentals -> FME Architecture -> FME Features -> fme_type
 

ATTR_TYPE_MAP

This section defines how a format's attribute types map into fme attribute types.

A good starting point for the ATTR_TYPE_MAP would be:

 ATTR_TYPE_MAP  char(width)             fme_char(width)            \                char(width)             fme_varchar(width)         \                string                  fme_buffer                 \                datetime                fme_datetime               \                time                    fme_time                   \                date                    fme_date                   \                real64                  fme_real64                 \                real32                  fme_real32                 \                int32                   fme_int32                  \                logical                 fme_boolean                \                int16                   fme_int16                  \                number(width,decimal)   fme_decimal(width,decimal)


The right-hand side lists all the possible attribute types that FME understands. It is important that each of these types have a mapping into the specific format attribute types used by your format. The IFMEReader::readSchema() method should return the attribute types on the left hand side on feature types that have user attributes.

Specific format attributes that contain numbers in parenthesis (e.g. char(20), number(10,10)) are only used for mapping fme types to format specific types. Thus

number(31,15)       fme_real64

will not have any effect if this is the source format.

In particular, it's important to get the date fields handled correctly -- it's good if you know that something is a date or a datetime, because then when we go out to a database we will generate the correct output types. However there is some responsibility that goes with this -- if you say something is a date or datetime, you need to format the attribute value correctly – YYYYMMDD, or YYYYMMDDHHMMSS – that is the official FME way for storing and exchanging dates and times.

It's possible that your format doesn't support one or more of the fme attribute types. In this case, map it to an appropriate attribute type your format does support. For example, you mapping might look like Shape's/DBF's (dbftypes.fmi):

ATTR_TYPE_MAP char(width) fme_char(width) \ char(width) fme_varchar(width) \ $(DBF_TYPES) \ char(255) fme_buffer \ char(20) fme_datetime \ char(12) fme_time \ date fme_date \ number(31,15) fme_real64 \ number(31,15) fme_real32 \ number(11,0) fme_int32 \ logical fme_boolean \ number(6,0) fme_int16 \ number(width,decimal) fme_decimal(width,decimal)

Notice that fme_buffer, fme_datetime, and fme_time all map to char(n) of some kind, and that both 32 & 64 bit floating point numbers map to number(31,15).

 


In general the format attribute types should not be prefixed by some format identifier, unlike with the GEOM_MAP entries.


For formats that don't have the concept of fixed vs. variable length strings, you should preferentially map strings to fme_varchar ahead of fme_char. This means automatic translations to database formats will use varchar columns instead of char columns, unless the source format knows otherwise.

IMPORTANT NOTE: Format specific attributes MUST use the format's own attribute types (types on the left hand side of ATTR_TYPE_MAP).

Incorrect use:

WORKBENCH_EXPOSABLE_ATTRIBUTES \myformat_name   fme_char(10)   

Correct use:

WORKBENCH_EXPOSABLE_ATTRIBUTES \myformat_name   char(10)

If the right side has a (width) or (width, decimal) and you cannot/do not want to use these parameters then use a hyphen on the left side.

double-                fme_decimal(width,decimal)

Essentially the hyphen means that the width, decimal will be thrown away.

It is important that a format has a full mapping for the generic fme attribute types, for if Workbench cannot map a generic type (say, fme_char), bad things will happen.

Feature Type Name Constraints

Following metafile directives determine how feature type names are handled.

Assumptions:

FEATURE_TYPE_CASE: This determines whether the attributes should be all upper case, lower, case, or it doesn't matter. Only affects writer. Possible values are:

  • ANY - Name will be left alone
  • UPPER - Name will be forced to upper case including any non-alphabetic characters. For example accented characters are non-alphabetic but may have upper and lower case version.
  • LOWER - Name will be forced to lower case including any non-alphabetic characters.
  • FIRST_UPPER - First character of the name will be forced to upper case including any non-alphabetic characters.
  • FIRST_LOWER - First character of the name will be forced to lower case including any non-alphabetic characters.
  • UPPER_FIRST_ALPHA - Force name to uppercase and then ensure that first character is an alphabet (A-Z). Accented characters are not treated as alphabets.
  • ANY_FIRST_NONNUMERIC - Ensure that the first character is not a number, if it is then put an underscore in front of the name.

FEATURE_TYPE_LENGTH: This determines the max length of a feature type name in characters not bytes. FEATURE_TYPE_INVALID_CHARS: This determines which characters are invalid for a feature type name. FEATURE_TYPE_RESERVED_WORDS: This determines words that have special meaning for the format and cannot be used as feature type names. Value of this directive is space delimited list of words. See sqlreservedwords.fmi for an example. FEATURE_TYPE_LENGTH_INCLUDES_PREFIX: This directive does not take any value but we provide a dummy value just so that it doesn't mess up where we expect name value pair when this directive is actually present in the metafile. Just the presence of this directive indicates that the maximum length of feature type will be inclusive of the schema or user prefix usually for database formats. "." is treated as part of the user/schema name. e.g. A feature type name could be dev.myTables. If this directive is present then whole of dev.myTables will be used to determine the maximum allowed length otherwise only myTables will be used for maximum length calculation. Example: In most of our database metafiles we use the following @TCL function call to apply the feature type name constraints.

 @TCL("set no_schema_prefix yes; set invalid_chars_regexp {./*;:!#&-\"'$[]}; set max_length 64; source {$(FME_MF_DIR_UNIX)/$(tclFileNameToUse)}")

The above line can be replaced by following metafile directives

 FEATURE_TYPE_LENGTH_INCLUDES_PREFIX dummyValue FEATURE_TYPE_INVALID_CHARS ./*;:!#&-\"'$[] FEATURE_TYPE_LENGTH 64 FEATURE_TYPE_CASE ANY INCLUDE formatReservedWords.fmi

INCLUDE line above is to keep the length of metafile reasonable. formatReservedWords.fmi will look like

 FEATURE_TYPE_RESERVED_WORDS    \   ABSOLUTE                     \   ACCESS                       \   ACCOUNT                      \   ACTION                       \   ACTIVATE                     \   ADD                          \   ADMIN                        \   AFTER                        \   ALL                          \   .   .   .

Attribute Name Constraints

There are a few directives that determine how attributes are handled. These are used by both Workbench and the Universal Translator (UT).

ATTRIBUTE_CASE: This determines whether the attributes should be all upper case, lower, case, or it doesn't matter. This only affects the writer. Possible values are:

  • ANY - Name will be left alone
  • UPPER - Name will be forced to upper case including any non-alphabetic characters. For example accented characters are non-alphabetic but may have upper and lower case version.
  • LOWER - Name will be forced to lower case including any non-alphabetic characters.
  • FIRST_UPPER - First character of the name will be forced to upper case including any non-alphabetic characters.
  • FIRST_LOWER - First character of the name will be forced to lower case including any non-alphabetic characters.
  • UPPER_FIRST_ALPHA - Force name to uppercase and then ensure that first character is an alphabet (A-Z). Accented characters are not treated as alphabets.
  • ANY_FIRST_NONNUMERIC - Ensure that the first character is not a number, if it is then put an underscore in front of the name.

ATTRIBUTE_LENGTH: This determines the max length of an attribute ATTRIBUTE_INVALID_CHARS: This determines which characters are invalid in an attribute name. ATTRIBUTE_CASE_IGNORE_LIST: This sets a list of attributes that are not to be affected by the ATTRIBUTE_CASE directive DEST_ILLEGAL_ATTR_LIST: This determines words that have special meaning for the format and cannot be used as attribute names. Value of this directive is space delimited list of words. See sqlreservedwords.fmi for an example.

It's possible for the value of the directive to depend on whether the format is being read or written. In this case you can check the direction within the metafile to ensure the correct value is used. The following is an example taken from geodatabase_sde.fmf:

 INCLUDE [if {"$(FMEGEN_DIRECTION)" == "Destination"} { \              puts {ATTRIBUTE_CASE UPPER};              \              puts {ATTRIBUTE_LENGTH 31};               \          } else {                                      \              puts {ATTRIBUTE_CASE ANY};                \              puts {ATTRIBUTE_LENGTH 255};              \          } ]

META_TABLE <TABLENAME> <ACCESSTYPE> [<TABLEVALUE>]+

These statements are used to define tables whose values are to be placed in correlation or definition lines. Each time the table is referenced the next table value is used. In other words, the table value is incremented as each feature is processed. This is done to ensure that definition lines always match corresponding correlation lines. The next value is retrieved from the table using the {FME_TABLE:<tableName>} substitution value. Currently the <accessType> is stored but not used. The idea would be to support values such as GROUP_RELATED, SEQUENTIAL, RANDOM, etc. If GROUP_RELATED is specified then the value returned is associated with the group. Each group Name is thus associated with an index into the table. This should be specified if the table value must appear on both an FME mapping file definition line and a FME correlation line and the values must be in sync.

The common use of this is:

 META_TABLE ALLOWED_OUTPUT_FEATURE_TYPES GROUP_RELATED [<FEATURE_TYPES>]+

When it's used this way, you'll need the line

 DEF_LINE_BREAK NO_DEF NO_ATTRIBS

as when it was set to "ATTRIB_CHANGE" the features were duplicated in an OBJ->OBJ translation and in a translation from Shape->OBJ where the shape feature type wasn't what was specified in [<FEATURE_TYPES>]+ caused an error to occur on workspace generation (although that could be related to WORKBENCH_CANNED_SCHEMA). Also, specify:

 WORKBENCH_CANNED_SCHEMA <schema file>

The TFS Fixed Schema Plug-in is a good example of how this all works.

Single Destination Feature Type Usage
Another common usage is for writers like GENERIC (in dynamic mode), 3DS and OBJ. In these three cases, the destination dataset has a single workbench node whose actual feature type and attributes are irrelevant to the writer at runtime. At file generation time, however, one would want all source feature types to feed into the single destination type.

There are three points of configuration to set this up in the metafile:

  1. A META_TABLE command is added to allow only a single feature type to be emitted
  2. A canned schema is used to define the one feature type that will be emitted
  3. The DEF_LINE_BREAK is set so that all input is connected. The internals of FME will detect the case where META_TABLE is limiting the output to a single table, and will join all input feature types to the one output feature type at generation time.

In the metafile for 3DS, these three lines look like the following. The order is not significant, but they must be located in the appropriate sections of the metafile. The "NO_ATTRIBS" prevents the user from adding attributes to the destination feature type.

META_TABLE ALLOWED_OUTPUT_FEATURE_TYPES GROUP_RELATED 3DS_ELEMENTWORKBENCH_CANNED_SCHEMA 3ds.schDEF_LINE_BREAK ATTRIB_CHANGE NO_ATTRIBS

This will generate workspaces with all input feature types joined to a destination feature type named "3DS_ELEMENT". This type is defined in schemas/3ds.sch as:

FEATURE_DEF 3DS_ELEMENT                                 \   fme_geometry{0}                             3ds_mesh \   fme_geomattr{0}.3ds_diffuse_color           string   \   fme_geomattr{0}.3ds_specular_color          string   \   fme_geomattr{0}.3ds_ambient_color           string   \   fme_geomattr{0}.3ds_material                string   \   fme_geomattr{0}.3ds_texture_image           string   \   fme_geomattr{0}.3ds_texture_map_blur        string   \   fme_geomattr{0}.3ds_texture_map_scale       string   \   fme_geomattr{0}.3ds_texture_map_offset      string   \   fme_geomattr{0}.3ds_texture_map_rotation    string   \   fme_geomattr{0}.3ds_texture_map_mirror      string   \   fme_geomattr{0}.3ds_texture_map_intensity   string

Correlation Line Directives

CORR_LINE_TEMPLATE defines the format of the correlation line. It is of the form

CORR_LINE_TEMPLATE <groupName> [<def_attr> <def_attr_value.]*

For formats with only one format-specific type, this is typically:

CORR_LINE_TEMPLATE {FME_GEN_GROUP_NAME}

For formats with multiple format-specific types, this is typically:

CORR_LINE_TEMPLATE {FME_GEN_GROUP_NAME} <format-specific-type-attribute> {FME_GEN_GEOMETRY}

The second form is needed on both the source and destination sides whenever the format has more than one format-specific type. If you leave this out on the reader side, you can’t successfully convert to a format that splits the data up by geometry type (e.g. Shape). If you leave this out on the writer side, the writer won’t get the format-specific type. These bug symptoms only occur if you are using the Universal Translator or generated mapping files. Therefore, it is good practice to test each format by converting the alltypes MIF to the format, and then converting the format to Shape, all via the Universal Translator.

FORMAT_TYPE

This required directive must be one of the following values: BUILTIN, DYNAMIC, WRAPID, JAVA, PYTHON.

This line will prevent FME from iterating through each type when creating the reader or writer. This allows more specific errors to appear in the logfile, and prevents misleading log messages from appearing should FME fail to create this reader or writer.

Note: The existence of the keywords DYNAMIC_LIBRARY implies FORMAT_TYPE DYNAMIC

FORMAT_TYPE_READ and FORMAT_TYPE_WRITE allow you to denote a format type specific to the reader or writer.

DYNAMIC_LIBRARY

Name of actual dll (minus the extension) to use for the reader/writer relative to FME_HOME/plugins/ and FME_HOME/ (In that order). This is used to make sure that we find the 3rd party DLL in the plugins directory, do not clutter the FME install directory and keep related files together.

If you're wanting to put the dll in a folder within the plugins directory (like PDF does), then your line will look like:

 DYNAMIC_LIBRARY <folder name>/<dll name minus file extension>

DYNAMIC_LIBRARY_READ and DYNAMIC_LIBRARY_WRITE allow you to denote a .dll specific to the reader or writer.

Re-using entry points from an existing plugin

If you are adding a new reader (or writer), which piggybacks off an existing format, and this existing format the writer DLL entry point FME_createWriter() (or FME_createReader()) then add the following line in the metafile to inform our FORMAT reader that there is no corresponding writer:

 DYNAMIC_LIBRARY_WRITE __NOWRITER__

If you're a writer only, then you'll need to add the line:

 DYNAMIC_LIBRARY_READ __NOREADER__

FORMAT reader will then provide the value NO to the attribute CAN_CREATE_[READER|WRITER].

PLUGIN_NAME

The actual plugin name for python. This specify the module to load.

PLUGIN_NAME_READ and PLUGIN_NAME_WRITE allow you to denote a plugin name specific to the reader or writer.

FME_PYTHON_MINIMUM_VERSION

To specify the minimum python version inside the metafile.

 FME_PYTHON_MINIMUM_VERSION 2.6

FME_PYTHON_PATH

Specify a python path inside the metafile.

 FME_PYTHON_PATH "$(FME_HOME)../formats/myformat/"

COORDINATE_SYSTEM_GRANULARITY

This directive defines the lowest level of coordinate system granularity supported by a format. Valid values are:

  • DATASET: This is the old behaviour where all features sent to a writer will have the same coordinate system. The very FIRST feature with coordinate system (it may not be the very first feature sent to the writer) was used as the reference coordinate system for all the subsequent features. If any feature had a different coordinate system it would then be reprojected to the reference coordinate system. If you don't specify the COORDINATE_SYSTEM_GRANULARITY directive, this is assumed by default. But if you do specify this directive explicitly in the metafile then a slightly modified approach will be taken to get the reference coordinate system. Instead of using the first feature with coordinate system, FME would use the first feature's coordinate system as the reference coordinate system. This mean that if the first feature doesn't have any coordinate system specified then subsequent features that may have a coordinate system will never be reprojected.
  • FEATURE_TYPE: This will ensure that the same coordinate system will be set for a given feature type. Again, the coordinate system on the first feature for a given feature type will be used as the reference coordinate system. All other subsequent features will be re-projected to the reference coordinate system if the reference coordinate system is not empty. Most database formats will be in this category.
  • FEATURE: This will ensure that FME core will never touch the coordinate system on the feature, it will be up to the writer to handle per feature coordinate systems. Most raster writers will be in this category.

Although this directive can be added anywhere in the metafile, it is strongly suggested that this directive be used within the DESTINATION_PREAMBLE section of the metafile to maintain backwards compatibility. Adding the directive in DESTINATION_PREAMBLE section will also ensure that the value of this directive will only apply to newly generated workspaces. Existing workspaces should work the same way as before this change.

IMPLEMENTS_FTR

This directive defines if the reader supports the FEATURE_TYPES directive when reading schema and data features. If a reader filters the features based on the FEATURE_TYPE directive natively, then reader core (STFReaderImp) doesn't have to do any filtering based on FEATURE_TYPES directive. WMS reader uses this directive because the FEATURE_TYPES list doesn't match with the features that are returned because of the hierarchical nature of the feature type naming scheme. E.g.

  $(FMEGEN_SOURCE_KEYWORD)_IMPLEMENTS_FTR yes

Metafile Directives Specific to Workbench

_WORKBENCH

This is a source and destination macro set in the control file object, which is accessible in metafiles with values 'yes' or 'no'. It is intended to indicate two things: that the metafile usage is currently for workbench, and that the metafile usage is currently for generate.

WORKBENCH_DEFLINE_PARMS

This section gives more information about the DEF line parameters in the DEF_LINE_TEMPLATE. Each parameter can supply a GUI line and a default value. This defines how the GUI will appear under the 'Parameters' tab in the feature type properties dialog.

WORKBENCH_EXPOSABLE_ATTRIBUTES

This gives a list of all the attributes that Workbench will show under the 'Format Attributes' tab in the feature type properties dialog, and the attributes' types.

WORKBENCH_ADDITIONAL_ATTRIBUTES

NOTE: This keyword has been deprecated. Use [FORMAT_PARAMETER WRITER_AUTO_EXPOSE] for automatically adding attributes to destination feature types.

This gives a list of attributes that will, by default, be added to all feature types a given reader returns. This is usually used in combination with the WORKBENCH_EXPOSABLE_ATTRIBUTES tag to automatically expose format attributes. It has the same syntax as WORKBENCH_EXPOSABLE_ATTRIBUTES.

WORKBENCH_CANNED_SCHEMA

This points to a schema file (the .sch files in install/schemas) defining all the canned schemas that will end up as destination feature types for this format. This directive is only applicable to writers. See the GXF metafile (gxf.fmf) for an example.

Alternatively, instead of a .sch file, this directive can also be given the value READ_AS_SOURCE which indicates that the writer should use the reader readSchema call to determine its schemas. This can be used in cases where there are several fixed schemas to be chosen from dynamically based on input. This is a similar but more dynamic way of defining a set fixed schemas for a writer without making static .sch files.

When this directive is specified, then the directive WORKBENCH_NO_FEATURE_TYPE_FANOUT must also be specified because fan-out does not make sense in a fixed schema format.

WORKBENCH_NO_FEATURE_TYPE_FANOUT

By default, Workbench assumes your format will support feature type fanout. Your format will get this for free if it is a file-based format. If it is directory-based, or a database format, your format will require special code. If you have a directory or database format that does not support feature-type fanout, put this stand-alone directive in your metafile. Workbench will then disable the fanout option in the feature type properties dialog.

WORKBENCH_ATTR_INDEX_TYPES

This specifies the types of indexes that attributes on this format can have. If this is specified, an extra column is provided in the feature type properties dialog for this format, allowing the user to select indexes for specific columns. The syntax looks like this:

WORKBENCH_ATTR_INDEX_TYPES <type1> <type2> <type3>

If the user selects an index for an attribute, it appears after the attribute's type on the feature type's DEF line, separated by a comma. For instance:

SPUD_DEF MyTable \  column_one  varchar(255) \  column_two  varchar(10),indexed

FORMAT_PARAMETER

This is a generic way of specifying reader/writer parameters for use in Workbench. The way you specify such a parameter is like this:

FORMAT_PARAMETER <key> <value>

The key is completely arbitrary. Using FORMAT_PARAMETER, instead of making a whole new directive (as the ones above), allows you to make new metafile parameters instantly, without having to modify about 10 C++ classes. Any parameters specified in this way will automatically be stored in workspaces and be usable in Workbench. It is anticipated that most new reader/writer configuration directives will be done this way.

FORMAT_PARAMETER AUTO_PUBLISH

Specifies which reader/writer directives should be published when added to a workspace. If this is not specified, the default behaviour is to publish the source and destination dataset parameter.

NOTE : If this is used in a metafile to auto publish some parameter(s) then ensure that SourceDataset and DestDataset are also included in the list for auto publishing. Syntax:

FORMAT_PARAMETER AUTO_PUBLISH <macro1> <macro2> ...

<macro1> <macro2> are the macro names used in the reader/writer preambles. These are the macros that will be published.

FORMAT_PARAMETER ADVANCED_PARMS

Specifies which reader/writer directives should be shown under the 'Advanced' branch, instead of in the main 'Parameters' branch in the navigation tree. Syntax:

FORMAT_PARAMETER ADVANCED_PARMS <macro1> <macro2> ...

<macro1> <macro2> are the macro names used in the reader/writer preambles. These are the macros that will be shown under 'Advanced'.

Each format can only have one ADVANCED_PARMS list, so the reader and writer have to share a list.

FORMAT_PARAMETER READER_AUTO_EXPOSE

Specifies a list of attribute name and attribute type pairs, to be automatically exposed on source feature types.

Syntax:

FORMAT_PARAMETER READER_AUTO_EXPOSE <attrName1> <attrType1> <attrName2> <attrType2> ...

Example usage:

FORMAT_PARAMETER READER_AUTO_EXPOSE \fme_basename    char(50)            \igds_level_name char (255)

FORMAT_PARAMETER WRITER_AUTO_EXPOSE

Specifies a list of attribute name and attribute type pairs, to be automatically exposed on destination feature types.

Syntax:

FORMAT_PARAMETER WRITER_AUTO_EXPOSE <attrName1> <attrType1> <attrName2> <attrType2> ...

FORMAT_PARAMETER ALLOW_DATASET_CONFLICT

Specifies workbench to ignore dataset conflict warning, if a format can read from one dataset and write back to same dataset. If this is not specified, the default behaviour is to is to warn user if source and destination datasets are same. Syntax:

FORMAT_PARAMETER ALLOW_DATASET_CONFLICT [YES|NO]

FORMAT_PARAMETER AUTO_FANOUT_ATTRIBUTE

Specifies workbench to use the "attributeName" specified as the attribute to use for setting the fanout property of destination dataset during workspace generation, adding new dataset or adding new feature type. For now the attribute type is assumed to be char(50), but currently this is not used. Syntax:

FORMAT_PARAMETER AUTO_FANOUT_ATTRIBUTE <attributeName>

Miscellaneous

Value Matching

Matching between different attribute values must also be handled. A good example of this is the matching of colors from one system to another. Clearly it is not possible to explicitly match colors since a system like MapInfo specifies 16 million different colors. Instead it is proposed that the FME support 64 different colors. 4 bits for blue, 4 for red, and 4 for green. The mapping of colors from other systems to these is accomplished by dividing the colors appropriately. For example 24 bit color is converted to FME color values as follows:

MapInfo Green Component / 16MapInfo Red Component / 16MapInfo Blue Component /16

Hopefully this mapping can be accomplished using @Evaluate(). If not then another alternative will have to be done. Of course systems which have 64 or fewer defined colors can be directly mapped. It is important to note that we cannot merely divide the entire color value by a single value to get the same effect as the colors at the bottom end of the value will be unfairly discriminated against.

Predefined Macros

A number of predefined macros are required in order to specify information properly. These macros are defined and can be referenced throughout FME meta files.

$(FMEGEN_SOURCE_KEYWORD):

  • This macro expands to the source keyword which will be used throughout the resulting mapping file.

$(FMEGEN_DESTINATION_KEYWORD):

  • This macro expands to the source keyword which will be used throughout the resulting mapping file.

Substitution Values

These values (aside from being poorly named) are used to specify that you want the current group name to be substituted into a definition line or correlation line. It is expected that these will be used primarily for the specification of definition lines and correlation lines.

  • {attrName}:
    • This is used to extract an attribute value from the schema generation feature and store it within a definition line or correlation line. This is currently designed for use on a definition line and removes the referenced attribute from the feature once it is substituted. See the SDEdefs.txt file for an example of how to use this functionality. It is used in the DEF_LINE_TEMPLATE statement.
  • {FME_GEN_GROUP_NAME}
    • This macro contains the current name of the group as defined by the current source feature which is being used to drive the mapping file generation. This value is useful for defining definition and correlation lines within a meta file.
  • {FME_GEN_GROUP_BASE}
    • This macros is the same value as the GEN_GROUP_NAME above except that if the group name contains any file extension names then the file extension is removed. For example in MOEP 83d088d.bin would be output as 83d088d for {FME_GEN_GROUP_BASE} and 83d088d.bin for {FME_GEN_GROUP_NAME}
  • {FME_GEN_GROUP_NUMBER}
    • This is not supported as the same functionality can be performed using the {FME_TABLE:<meta table name>} directive with more flexibility. See igdsdefs.txt for example of how this is done.
  • {FME_GEN_GEOMETRY}
    • This value contains the current geometry value of the current definition line in the form consistent with the format which the value is specified.
  • {FME_TABLE:<meta table name>}
    • This value is substituted by the current value in the meta table identified by <meta table name>.

FAQ

What is the purpose of the GENERICIZE_GEOMTYPE_ATTR_NAME line in a metafile? Nothing seems to break when I commented that line out of the sample, as long as I have the generic format attribute listed on the CORR_LINE_TEMPLATE line.

That directive was only put there so that WHEN the CORR_LINE_TEMPLATE does not list it (which in many situations it does not) there is a way of specifying the attribute. So, while there's no harm in asking for it always to be listed, it decouples the two situations and makes things cleaner. GENERICIZE_GEOMTYPE_ATTR_NAME was added later, and I only put it into metafiles where CORR_LINE_TEMPLATE didn't have what I needed (so the FME basically can get GENERICIZE_GEOMTYPE_ATTR_NAME from CORR_LINE_TEMPLATE, preventing a need to update all existing metafiles).

For the plug-in builder, you should always specify the GENERICIZE_GEOMTYPE_ATTR_NAME line.


What is the meaning of '__'?'

This is used to insert a return line in the generated mapping file. The characters are only used in preamble declarations.

Is it possible to add a line to a metafile, so that the FME_MINIMUM_BUILD is added to the generated mapping file?

Yes, you can put the statement into the "PREAMBLE" section of your metafile. Example:

 PREAMBLE FME_MINIMUM_BUILD 999 END_PREAMBLE


What are essential GEOM_MAP lines in a metafile?

This depends on your format. If you're thinking minimal, you probably should look to the textline format. Here are its lines

    • GEOM_MAP text_line_none fme_no_geom
    • GEOM_MAP text_line_none fme_point
    • GEOM_MAP text_line_none fme_line
    • GEOM_MAP text_line_none fme_polygon
    • GEOM_MAP text_line_none fme_text
    • GEOM_MAP text_line_none fme_ellipse
    • GEOM_MAP text_line_none fme_arc
    • GEOM_MAP text_line_none fme_rectangle
    • GEOM_MAP text_line_none fme_rounded_rectangle

This maps all fme geometries to none, and maps the text_line_none geometry to 'fme_no_geom'


Are ATTR_MAP and GEOM_MAP defining mappings between types? Then what do the many to many mappings mean?

Order matters. The first matching element from the relevant side is matched to its opposing side. So an fme_geometry is matched to text_line_none but text_line_none is mapped only to fme_no_geom.


What other essential lines are there in metafiles? (i.e. that all formats should have)

Here are three that can be easily overlooked

  • ATTRIBUTE_CASE ANY
  • ATTRIBUTE_LENGTH some_number. If you don't have a limit on your reader/writer, use the value 255.
  • ATTRIBUTE_INVALID_CHARS "@&" These are simply no good for FME. If you don't include this directive, FME assumes the default set of invalid chars "./*;:!#&-\"'$"