The Problem
Recently, one of our customers gave us a raster that looked absolutely black in the FME Viewer:
Closer inspection shows that the pixels have a good variety of values, so why are the contents not visible?
Let's have a closer look at the information window under "Raster Point Info" - we see that we have an image consisting of three 16 bit bands, what defines an RGB48 interpretation. This means that the range of available values for each band goes from 0 to 65535. The actual values on our image do not exceed a few hundred.
In other words, this raster contains 8-bit values stored in 16-bit bands.
The completely black display is partly due to how the FME Universal Viewer interprets the data; 0 is black and 65535 is white, so even a value of 256 is not going to register very highly. But, really, this is a problem with the data. We need to set the correct interpretation on the existing values or adjust the values to match the current interpretation.
Solution 1: RasterExpressionEvaluator
One possible solution is to use the
RasterExpressionEvaluator. We can apply the following band list and expression:
RED16;GREEN16;BLUE16
A[0]*200;A[1]*200;A[2]*200
However, this requires some knowledge of this advanced transformer and a few experiments to find an optimal multiplier.
Solution 2: RasterInterpretationCoercer
Another solution is to use the
RasterInterpretationCoercer, which may be a better choice for this situation. It contains four conversion options, giving users much better control over the whole process.
Conversion options
The conversion options include:
-
Cast
-
Bounded Cast
-
Scale by data values
-
Scale by data type
If we change the interpretation of our raster from RGB48 to RGB24 using 'Scale by data value' option, we will get the following nice and clear image:
Why did I pick this option, and how do these options work?
You can find the explanation how these options work in the transformer's documentation, but honestly speaking, it doesn't look clear enough to me - I don't have C-programming or, more general technical background to understand easily "overflows and roll overs" of the values, so I decided to make a few experiments for myself and show results for those who prefer a good visual explanation.
I will go through all the options and show how they work, and how you can pick the right one. For this purpose, I prepared a simple contour file and made a profile (brown, made with ProfileBuilder) along the red line:
Based on contour data, I generated a one band DEM raster and made two tiles. The
DEMGenerator makes Real64 rasters, and I am going to try to make it UINT8 - I know that rasters can be really big and want to save on disk space.
Now, before we continue with the conversion modes, let's have a look at the parameters. There are four of them where we pick the conversion mode. Why do we have so many? That's because Workbench has no way of knowing ahead of time what type of rasters will be passed through this transformer - whether it will be a color image (e.g. RGB24, RGBA64, Green16, etc.) or a numeric raster (Real64, Int32, etc.), and our destination can be either color or numeric. All possible conversion combinations are listed as separate parameters (we probably could have just two conversion parameters ("From Color" and "From Numeric") but for backward compatibility we had to have four). Based on our choice in "Destination Interpretation Type" only two of four parameters can be enabled.
Let's try to send one of the tiles through the transformers using "Cast" method:
Before
After
Inspection of different areas shows that the image (now UINT8) matches the source DEM (the only difference is that the values were rounded), so it looks like "Cast" is a pretty good option.
Let's have a closer look at the second tile:
Before
After
That's definitely wrong. For better understanding of what is going on let's build a profile of the new DEM (yellow) and compare it with the original profile (brown):
The elevation gradually goes up until it reaches 255 meters, which is the maximum value possible for for the type UINT8, and then abruptly it drops down to 0, after that, this pattern repeats again. The new elevation can be expressed with the following formula:
elevation % range_for_the_bit_depth = cast_elevation
or, if we take the elevation of the red dot on our UINT8 image:
575 % 256 = 63
So, the "Cast" option does not allow us to get a good conversion for the entire dataset.
"Bounded Cast" works nicely for the left tile. For the right tile it makes something different, but it is hardly any better - the top of the hill is simply chopped off (that is, everything above 255 meters) and everything below 0 is set 0:
Now we see the limitation of "Cast" and "Bounded Cast". They will work well as long as the values of the raster do not exceed the chosen destination interpretation type. As long as you know for sure that all your data is below the maximum value for the interpretation, these options are a good choice, otherwise, they screw up both pixel values and visual representation.
To avoid this, consider another interpretation type with higher bit depth or another interpretation option - we still have two more to try out.
The option "Scale by Data Value" makes the tiles that look nice and very close to the originals.
Before
After
However, the elevations on both tiles are scaled - expanded on one tile and shrunk on another:
The graphical representation of this conversion type could look as follows:
The range between minimum and maximum values found in the source is scaled to the full range of available values of the destination type. The tiles in the example have ranges 25 to 200 (left tile) and -45 to 575. Both resulting rasters have values scaled to the range between 0 and 255.
This option won't keep the true representation of the source raster, however, it can be useful for making color images:
The image above was made by converting Real64 to Red8, and then two extra bands were added - Green8 set to 100, and Blue8 set to 50. I also added vector contours on top of the image.
The last option, "Scale by Data Type" gives us a simple gray raster with all the pixels having the value of 128.
The full range of the source type is scaled to the full range of the destination type. Because the type of Real64 has a range between ~-1.78e+308 and ~1.78e+308, but the actual values vary from -45 to 575, all our destination values are squeezed into a tiny range somewhere very close to 128.
On the other hand, if the values are more or less evenly spread through the entire range, this method can help preserve visual representation:
Before
After
To summarize:
The first two options, "Cast" and "Bounded Cast" are better for keeping the original values (or rounded if we go from Real to Integer) under the condition that the destination range is big enough to fit all the values. These options are useful for Numeric to Numeric conversions.
The last two options, "Scale by Data Value" and "Scale by Data Type" help preserve the visual representation and are useful for Numeric-to-Color or Color-to-Color conversions.
As for my tiles, I have to pick another interpretation because I want to keep the correct elevation. INT16 seems to be a reasonable alternative because even if my project covers the entire world, it still won't exceed the
maximum and the
minimum (in meters) we can find on our planet.
The attached template, RasterInterpretationCoercer.fmwt, shows all the options explained above.
RGBA to RGB Option
The "RGBA to RGB" option tells the transformer what to do if we convert an RGBA image (an image with three color bands and an alpha band responsible for the transparency) into a simple RGB (color) image, that is, if we remove the alpha band.
Here is an RGBA32 image - the portion outside the area of interest is fading gradually into full invisibility:
For making the picture visually clearer, I opened it in
Paint.Net, a nice free graphic editor. Our own Viewer does not show alpha transparency at this moment (but I hope it will soon).
As you can see, the alpha value within the red circle at the bottom of the image is equal to 114 and the area underneath is semitransparent - we can see through the contents the checkerboard pattern of the background.
If we use "Apply the Alpha Band" option, we still will be able to see the gradual disappearance of the contents, although the image now has the interpretation RGB24 (Paint.Net always shows alpha band, but all the pixels have the value of 255):
"Drop Alpha Band" makes a small miracle - it removes the alpha band and reveals what was hidden by the transparency of the alpha pixels:
Note how the other bands (Red, Green and Blue) behave in both situations - "apply" option changes them ("alpha influence"), "drop" leaves the bands unchanged ("no alpha influence").
The workspace and the RGBA source image are attached as AlphaHandlingOption.fmwt