The Fastest Way to Extract Values from Geospatial Data
A considerable part of what we do at Terramonitor can be summed up by the extract, transform, load (ETL) paradigm. This happens both in internal processes and in those that are externally triggered. And sometimes those processes are too slow.
A considerable part of what we do at Terramonitor can be summed up by the extract, transform, load (ETL) paradigm. This happens both in internal processes and in those that are externally triggered.
- Internally, when we acquire raw remoting sensing data from a satellite data provider and store preprocessed data in our own database.
- Externally, when a client requests any part of that data via our user interface, data API or their own GIS client.
The internal process is offline and not very time critical. The focus is in quality of data, and optimization will generally be in the order of reducing the time a process takes from 12 hours to 2. This saves computation costs, but any process taking hours is still best described as "leave it running and get back to it". The external process is online and is very time critical. It has to happen faster than a user can change the active tab in their browser. In this case, there is a big difference whether user response happens in 12 seconds or 2.
Cover Photo by Markus Spiske on Unsplash
The Three Iterations of Land Cover API
Our Land Cover API allows the user to query information for a disc-shaped area from any point on the globe, and receive a textual answer consisting of remote sensing data related to said area. The data to fetch is stored as GeoTIFFs in a cloud storage. The following three iterations are what we went through to get where we are, from slowest to fastest.
I – naïve implementation
Fully functional and very quick to implement, our first implementation was roughly as follows:
- download GeoTIFF from cloud storage
- run Python process and perform geospatial query with GDAL on GeoTIFF
- output pixel values
II – cloud optimized GeoTIFF
There was a lot of unnecessary data being transferred in the first implementation, as generally only a very small portion of a GeoTIFF is used per query. So the first improvement was optimizing data transfer.
- download only requested part of GeoTIFF from cloud storage using GDAL and cloud optimized GeoTIFF
- run Python process and perform geospatial query with GDAL on GeoTIFF
- output pixel values
III – hex dump (current)
The second implementation was fairly fast, but now the bottleneck was the Python process performing a geospatial query, which isn't even necessary as the downloaded data already contains only the requested information. And you don't need Python if you are just going to output the contents of a file. So we got rid of the Python process completely.
- download only requested part of GeoTIFF from cloud storage using GDAL and cloud optimized GeoTIFF
- transform GeoTIFF into a PNM image
- run od (hex dump file contents) to output pixel values
The earth observation company Planet released a blog post about reading a single pixel from a GeoTIFF file without using GDAL. In other words, building your own GeoTIFF reader. This requires deep understanding of the cloud optimized GeoTIFF format, including understanding the possible compression algorithms used.
We are happy to use gdallocationinfo for single pixel queries, and would happily use a similar GDAL binary for multiple pixels if one existed. Our current solution is not to read the contents of the GeoTIFF file at all, but transform it to a considerably simpler file format, and read that instead. In practice, this implementation is magnitudes faster than implementation I or II.
Processing Geospatial Data
If you are working with a large number of GeoTIFF files and are running into performance or hosting issues, please contact us for details on our implementation.
Contact us via a form or send an e-mail to sales@terramonitor.com