Dynamic Schema Components

This section describes the components and the configurations required to enable Dynamic Schema functionality.

The specific components are:

•  Merge Parameters (source feature type),
•  Dynamic Parameters (destination feature type), and
•  Workspace Resources (navigator panel).

Merge Parameters

A source feature type can be configured to output all features regardless of feature type name by setting the merge filter to asterisk (*).


 

Dynamic Parameters

A destination feature type can be configured to write all input features to a dataset with a schema determined at run-time (AKA: Schema Source).



There are two types of Schema Sources distinguished by there icons:

• Source Readers and
• Workspace Resources.

In the following screenshot, the schema_map reader is a Workspace Resource and the InteroperalisDatabase reader is a Source Reader.




The Source Readers are automatically available for use. The Workspace Resources must be manually defined.

Log File Messages

The feature types defined by each selected Schema Sources will be logged. As an example:
 

Reader `SHAPE_1' of type `SHAPE' provided schema for types: BusStops,CityParks,Roads  

If features are destined for a feature type and that feature type is not defined in the Schema Sources, then you will see the following type of error message near the bottom of the log file:

                       Features With No Schema defined
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Roads                                                                      863
==============================================================================
Total Features NOT Written                                                 863
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Workspace Resources

The Workspace Resources are readers that can be added to a workspace for the purpose of providing a schema definition for the writer. Any reader can provide the schema regardless of the output format of the writer (e.g. a Shapefile could provide the schema for a Geodatabase layer). You can add a Workspace Resource by right clicking 'Workspace Resources' in the Navigator panel and selecting 'Add Reader as Resource'. Once you are finished, the reader will now by accessible from the writer feature type > Dynamic Parameters > Schema Source Dialog.


 

Schema (From Table) Reader

The 'Schema (From Table)' reader is a special type of reader that can provide a schema based on a definition in a spreadsheet or database table.


The spreadsheet or table that you create to define the schema must have one row for each attribute definition with fields that provide the following information (the field names can be named as you like):
 

• Feature Type Name
  • So the writer knows which feature type should use the attribute definitions.
• Attribute Name
• Attribute Data Type
  • Possible values are:
    • fme_char(width)
    • fme_varchar(width)
    • fme_buffer
    • fme_datetime
    • fme_date
    • fme_time
    • fme_decimal(width,decimal)
    • fme_real32
    • fme_real64
    • fme_int32
    • fme_boolean
    • fme_int16
• Geometry Type
  • A semi-colon separated list of values representing the type of geometry provided.
  • A value only needs to be entered once per feature type in the row defining the first attribute for the feature type (see Attribute Order).
  • Possible values are the same as the possible values for the fme_type FME format attribute and are listed in the FME Fundamentals manual: http://docs.safe.com/fme/html/FME_Fundamentals/FME_Fundamentals.htm > Chapter 2 - FME Architecture > FME Features > fme_type
• Attribute Order
  • An optional number indicating the position of the attribute in the feature type.

In the image below you can see an example of a table that has the required fields (left) and the dialog for defining the Schema (From Table) reader (right).

Dynamic Schema Examples

The Dynamic Schema functionality provides the ability to determine the writer schema at run-time. This allows you to decouple your schema definitions from the workspace for greater flexability, reusabilty and less maintenance. It's important to know that the new dynamic schema must be implemented in more than one place. So to make it easy to start with this configuration, a parameter was added to the New Workspace Dialog named 'Dynamic Schema' (screenshot below), which will implement this configuration during the initial workspace generation.



Specifically, the Dynamic Schema functionality provides two options for determining the destination schema information at run-time:

• from a reader already in the workspace (Source Reader),
• from a reader added as a Workspace Resource (Workspace Resource Reader).

Each of these options will be discussed in the following examples by comparing a number of scenarios both before and after Dynamic Schema functionality existed. In each case, the result is fewer transformers, fewer feature types and no need to modify the feature types when the source or destination schema changes.

Download the Example Workspaces

1. These examples require the FME Training Dataset which is packaged with the FME Desktop 2011 Tutorial and Sample Dataset which can be downloaded from the following link:

• Download.

Important: once downloaded, ensure that you place the sample dataset in C:\FMEData

2. The Dynamic Schema Example files (workspaces, custom formats, CSV files, etc) can be downloaded from the following link:

• Download (see attached)
  

Schema from a Source Dataset (Example 1)

In this example, a workspace has been configured to read three feature types from an SDF dataset, reproject the features and then write the features to a shapefile dataset. If the schema of the source data (e.g. layer names, attribute names, geometry types) changes, then the destination feature types will need to be modified in the workspace.



With Dynamic Schema functionality, the writer schema can be determined at run-time by analyzing the reader's schema. So if the reader schema changes, the workspace doesn't need to.

Steps

The New Workspace Dialog will automatically configure this option when the "Dynamic Schema" option is selected.

Schema from a Template Dataset (Example 2)

In this example, a workspace has been configured to read three feature types from an SDF dataset, some of the attributes are renamed and then the features are written out to a shapefile dataset.

In this situation the attribute renaming (i.e. schema mapping) details are implemented as AttributeRenamer transformers is done in the workspace, but it is also common to maintain the schema mapping details external to the workspace and link the workspace to those details in order to determine the schema mapping at run-time. The SchemaMapper transformer is one example of how this can be done (see example 4).

If the schema mapping details are changed (e.g. attribute renaming), then the destination feature types will need to be modified in the workspace.




With Dynamic Schema functionality, the writer's schema can be determined at run-time by analyzing a workspace resource reader's schema. So if the schema mapping changes, the feature type properties don't need to.

Steps

1. Start with New Workspace Dialog and select the "Dynamic Schema" option,
2. in the Navigation Panel, right click Workspace Resources and select "Add Reader as Resource",
3. add the dataset that will provide the schema,
4. open the properties of the single writer feature type,
5. in the Dynamic Parameters section, click the Schema Sources button, un-select the source reader and select the workspace resource reader.

Schema from Details Contained Within a Dataset (Example 3)

This example starts with the same workspace as example 2. The workspace has been configured to read three feature types from an SDF dataset, some of the attributes are renamed and then the features are written out to a shapefile dataset.

Again, the schema mapping details are implemented in the workspace, but it is also common to maintain them external to the workspace (e.g. SchemaMapper).

Just like example 2, if the schema mapping details are changed (e.g. attribute renaming), then the destination feature types will need to be modified in the workspace.



With Dynamic Schema functionality, the writer's schema can be determined at run-time by reading a description contained within a dataset. As in example 2, the schema mapping can change, without the feature type properties needing to.

Steps

First you need to create a dataset that will provide the writer's schema definition. Here is an example:




Now that you have the dataset to provide the schema definition, complete the following steps:

1. Start with New Workspace Dialog and select the "Dynamic Schema" option,
2. in the Navigation Panel, right click Workspace Resources and select "Add a Reader as Resource",
3. add a Schema (From Table) reader that uses the dataset you created to provide the schema definition,
4. open the properties of the single destination feature type,
5. in the Dynamic Parameters section, click the Schema Sources button, un-select the Source Reader and select the Workspace Resource Reader.

Schema Mapping and Schema from Details Contained Within a Dataset (Example 4)

In the case where schema mapping details are stored outside the workspace, it is very likely that the writer schema will be stored externally as well. This scenario is the same as example 3, except that a SchemaMapper transformer is used instead of multiple AttributeRenamer transformers.

Just like example 2 and 3, if the schema mapping details (provided by the SchemaMapper transformer) change, then the writer feature types will need to be modified in the workspace.



With Dynamic Schema functionality, the writer's schema can be determined at run-time by analyzing a Workspace Resource reader's schema. So, now both the schema mapping details and the writer schema details can be created and maintained outside of the workspace.