Question

Building web services workspace to process JSON results from the Yahoo GeoPlanet web service.

Answer

Creating a workspace that uses a web service is quite straightforward and usually consists of three main parts:
  • getting user input;
  • building and submitting a URL, getting some response from the service;
  • exploring results and creating geometries - this part may vary significantly depending on the contents returned by the service.

Let's take Yahoo GeoPlanet web service and build a workspace that uses it. I will give step-by-step instructions showing the authoring process in Workbench.

Yahoo! GeoPlanet is "a resource for managing all geo-permanent named places on Earth. It provides the geographic developer community with the vocabulary and grammar to describe the world's geography in an unequivocal, permanent, and language-neutral manner."

This service can return place location as a point or a bounding box, place's parent, ancestors, siblings, neighbors, etc. For example, the parent of British Columbia is Canada, and Alberta is the sibling and neighbor. The service is accessed via HTTP GET method, which is supported by the [HTTPFetcher] transformer.

First thing Yahoo asks us to do is to get Application ID. The end-users of your workspace do not need to create it - they can use your own, however, there is a limit of 50,000 requests per day.

Now let's create a feature with no geometry and containing just one attribute CityName and give it the value "Port Moody". It's a small city in the Greater Vancouver area. Later this portion can be replaced with a reader containing features with geographic location attributes.
User-added image
The next step is building a URL.

The URL consists of two parts - the constant part, which the service address. In our case, it's http://where.yahooapis.com/v1/.

The second part supplies parameters to the web service - places.q() resource returns a collection of places. The last parameter is the Application ID. Check the API documentation for more information about resources and parameters. For the full list of possible parameters check the Yahoo GeoPlanet API

To combine the constant and the variable parts of the URL we use Concatenator, which gives us the following URL:

http://where.yahooapis.com/v1/places.q('Port Moody')?appid=qVRiFr7V34Fxy.1WCaagLud8Pv_HyREl9dHBpEZTeaoOKV4RLp3ZU9vt3uaK
User-added image
Generally speaking, before we submit the URL, we should make it safe, that is, we should not use characters that aren't allowed by URL specification:

"...Only alphanumerics [0-9a-zA-Z], the special characters "$-_.+!*'()," ,
and reserved characters used for their reserved purposes may be used
unencoded within a URL."

As you can see, we have a space between the words "Port" and "Moody". Most browsers as well as HTTPFetcher are smart enough to make the replacement of the spaces with the proper escaping sequence, which is %20, however this may not be the case with some characters and services.

For example, Google Earth doesn't like the pipe symbol (|). If you use Google Charts, where pipes are value separators, in Google Earth under Windows, you have to escape pipes with %7C.

The HTTPFetcher returns some response from the server in a form of an attribute (_url_contents by default):
<?xml version="1.0" encoding="UTF-8"?>
<places xmlns="http://where.yahooapis.com/v1/schema.rng" xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:start="0" yahoo:count="1" yahoo:total="1">
 <place yahoo:uri="http://where.yahooapis.com/v1/place/9225" xml:lang="en-US">
   <woeid>9225</woeid>
   <placeTypeName code="7">Town</placeTypeName>
   <name>Port Moody</name>
   <country type="Country" code="CA">Canada</country>
   <admin1 type="Province" code="CA-BC">British Columbia</admin1>
   <admin2 type="County" code="">Greater Vancouver</admin2>
   <admin3></admin3>
   <locality1 type="Town">Port Moody</locality1>
   <locality2></locality2>
   <postal type="Postal Code">V3H</postal>
   <centroid>
    <latitude>49.282429</latitude>
    <longitude>-122.829338</longitude>
   </centroid>
   <boundingBox>
    <southWest>
     <latitude>49.26997</latitude>
     <longitude>-122.927147</longitude>
    </southWest>
    <northEast>
     <latitude>49.330978</latitude>
     <longitude>-122.818733</longitude>
    </northEast>
   </boundingBox>
  </place>
 </places>

Depending on a service the response can be an HTML, XML, JSON, text, etc (if the service returns images, ImageFetcher should be used instead). Some services have parameters specifying the format. Yahoo GeoPlanet supports XML (default), JSON and GeoJSON. We can specify the format after the question mark before or after Application ID as format=json. This parameter will return the following result:
 
        {
 "places":{
  "place":[{
   "woeid":9225,
   "placeTypeName":"Town",
   "placeTypeName attrs":{
    "code":7},
   "name":"Port Moody",
   "country":"Canada",
   "country attrs":{
    "type":"Country",
    "code":"CA"},
   "admin1":"British Columbia",
   "admin1 attrs":{
    "type":"Province",
    "code":"CA-BC"},
   "admin2":"Greater Vancouver",
   "admin2 attrs":{
    "type":"County",
    "code":""},
   "admin3":"",
   "locality1":"Port Moody",
   "locality1 attrs":{
    "type":"Town"},
   "locality2":"",
   "postal":"V3H",
   "postal attrs":{
    "type":"Postal Code"},
   "centroid":{
    "latitude":49.282429,
    "longitude":-122.829338},
   "boundingBox":{
    "southWest":{
     "latitude":49.26997,
     "longitude":-122.927147},
    "northEast":{
     "latitude":49.330978,
     "longitude":-122.818733}},
   "uri":"http:\/\/where.yahooapis.com\/v1\/place\/9225",
   "lang":"en-US"}],
  "start":0,
  "count":1,
  "total":1}
}        

Note that I formatted the XML and JSON above for better readability.
User-added image
Now let's try to add a geometry to the feature based on the coordinates in both JSON and XML. JSONExploder is an easy way to get features (the request can return multiple results) and geometries (if any) out of the source attribute.

NOTE that the behavior of JSONExtractor and JSONExploder may change in 2009 release as they are undergoing significant upgrades.

The JSONExploder needs the following parameters:
User-added image
This will break the JSON text into features (in our case we have just one).

The extraction of attributes is accomplished by using JSONExtractor:

User-added image
This transformer will add the attribute latitude to the feature. We should use as many JSONExtractors as many attributes we would like to extract:
User-added image
If the service returns results in XML format, we search for the attribute values with StringSearcher (previously known as Grepper). The technique looks as follows:
 
  • StringSearcher finds the pattern and places the portion in parentheses into a list attribute _matched_part{}.
  • ListIndexer with index 0 forces the list attribute to become the main attribute _matched_part.
  • AttributeRenamer renames _matched_part to the name we are extracting (latitude, longitude, etc.)

So we need a set of three transformers for each attribute:
User-added image
In general, this procedure of XML reading is neither simple, nor reliable, nor universal. But the situation is changing with FME 2009! We added XQuery language to it. This makes search through XML much more effective. Check this article with XQuery examples (Article#: 000001454).

Now, after we got latitude and longitude, we can create a geometry for our feature with 2DPointReplacer
User-added image
After that, we can either simply send the feature to our Visualizer or write, for example, to Google_Earth_KML_(OGCKML) (Article#: 000001478) to check whether the location is correct: (See attachment: YahooGeoPlanetJSON.fmw).
User-added image
The point is placed in front of Port Moody City Hall and Theater.

The first workspace uses JSON, the second workspace shows XML/StringSearcher route.