[[PageOutline]] This page is part of the [wiki:Future MapGuide Future] section, where ideas are proposed and refined before being turned into RFCs (or discarded). Visit the [wiki:Future] page to view more! = Overview = This page is a living proposal for improving the scalability of the Tile Service. [[BR]] Mapguide supports tiled maps via both WMS and it's own Tile Service using GETTILETIMAGE. Recently support was added in Openlayers & Fusion to access the Tile Service via an API.[[BR]] [http://dev.openlayers.org/apidocs/files/OpenLayers/Layer/MapGuide-js.html OpenLayers. Layer. MapGuide][[BR]] [http://trac.openlayers.org/ticket/995 #995 (add support for MapGuide OS layer type) - OpenLayers - Trac][[BR]] = API Ideas = '''SETTILEIMAGE''' would allow the tile cache to seeded without needing to have remote access to the server file system. '''GETTILEPARAMS''' would return a list of scale ranges, tile sizes and bounds for a given tile cache. '''GETTILECACHE''' would return a 'map' of the tile cache, listing which tiles exist and which ones haven't yet been created '''HAVETILEIMAGE''' would enable a server to poll another server(s) in the cluster, [[BR]] returning status codes of 200 ok or 404 Not Found respectively. This allows the server to[[BR]] check the tile cache without triggering a tile render on the polled server. These would enable more extensive API based management of a cluster of Mapguide servers, via accurate seeding[[BR]] and management of the tile caches. = Cacheability by HTTP = [wiki:MapGuideRfc11] added support for a '''Stateless Http GETTILEIMAGE request''', however, these tiles are served [[BR]] without any cache headers which means they can only be proxied using a custom 'aggressive' proxy service. [[BR]] [ticket:467 Add proper cache headers & file date to GETTILEIMAGE response]. One of the the issues to be resolved is how long to set the cache headers to cache the tiles before checking back [[BR]] to the Mapguide Server. This could done with a serverconfig.ini default and then an overriding parameter in the [[BR]] ResourceHeader. == API additions to MgTileService == To support HTTP cacheability we need to be able to do the following via the MgTileService: a) Get timestamp information about a generated tile (to be able to apply expiry dates) b) "Peek" at the timestamp for a tile that may or may not be generated (to be able to compare against dates from if-modified-since headers) An API addition to MgTileService like the one below should be able to support the above scenarios. {{{ class MG_MAPGUIDE_API MgTileService : MgService { PUBLISHED_API: /// Returns the timestamp of when the tile for the specified map/group/row/col was generated. Returns NULL if no such tile exists /// MgDateTime* GetTileCreationDate(MgMap* map, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow); /// Returns the timestamp of when the tile for the specified map/group/row/col/scale was generated. Returns NULL if no such tile exists /// MgDateTime* GetTileCreationDate(MgResourceIdentifier* mapDefinition, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex); /// Returns the specified base map tile for the given map. If a cached tile /// image exists it will return it, otherwise the tile is rendered and added /// to the cache. MgTile* GetTile(MgMap* map, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow); /// Returns the specified base map tile for the given map. If a cached tile /// image exists it will return it, otherwise the tile is rendered and added /// to the cache. /// MgTile* GetTile(MgResourceIdentifier* mapDefinition, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex); }; }}} MgTile is defined like so. {{{ class MG_MAPGUIDE_API MgTile : public MgSerializable { PUBLISHED_API: /// Returns the tile image /// MgByteReader* GetImage(); /// Returns the date this tile was created /// MgDateTime* GetCreationDate(); }; }}} == MapAgent modifications == These API changes do not really affect the mapagent interface. From a calling client's perspective, they should still be able to send v1.2.0 GETTILEIMAGE requests with the same parameters and take advantage of HTTP caching behind the scenes. Behind the scenes the GETTILEIMAGE operation handler simply needs to check for the existence of a "If-Modified-Since" request header to take on a new code path. A rough pseudocode overview of this process would be {{{ if (ifModifiedSince header exists) { extract date from header call GetTileCreationDate if (tileCreationDate != NULL) { if (tileCreationDate is newer than header date) { call new GetTile write image from MgTile into result write date from MgTile into last modified response header write expires date a long period from that date (6 months? 1 year?) } else { set status code of 304. Write nothing into the result. Outer CGI/Apache/ISAPI handler is expected to handle this and write out the appropriate responses (see below) } } else { call new GetTile write image from MgTile into result write date from MgTile into last modified response header write expires date a long period from that date (6 months? 1 year?) } } else { call new GetTile write image from MgTile into result write date from MgTile into last modified response header write expires date a long period from that date (6 months? 1 year?) } }}} In our applicable CGI/Apache/ISAPI handlers, their additional responsibilities are to: * Pack any HTTP request headers into the request metadata of the MgHttpRequest before executing it. In the case of this API, look for if-modified-since * Handle the 304 internal status and write out any applicable response headers from the MgHttpResult that operations supporting HTTP cacheability (ie. The GETTILEIMAGE) should provide. MgHttpRequestMetadata and MgHttpHeader classes are not currently used in any of the existing CGI/Apache/ISAPI handlers. We should use them for this purpose. = Re-usable tile sets = Currently the concept of a tile set is tightly-bound to the Map Definition that severely limits its re-usability between other maps. The second problem is that because a tile set is bound to a Map Definition, we have the all too common problem of shotgun tile cache invalidation on even the most minute changes in the Map Definition and/or upstream dependent resources. == The TileSetDefinition resource == Ideally the tile set should be modelled not as a part of a Map Definition, but as a separate resource. A tentative schema for this resource would look something like this: {{{ Defines a tile cache Defines the parameters to access the tile cache A bounding box around the area of the tile cache The coordinate system as WKT used by the TileSetDefinition The display scales that the base map layers will have tiles available. Applies to the HTML viewer. A group of layers that is used to compose a tiled layer in the HTML viewer TileStoreParameters defines the parameters of this tile cache. The width of tile images in this tile cache The height of tile images in this tile cache The image format of tile images in this tile cache The tile image provider Collection of name value pairs for connecting to the tile image provider A type describing name and value pairs Text for the name of parameter Text for value of parameter Box2D encapsulates the the coordinates of a box in 2-D space Minimum x-coordinate Maximum x-coordinate Minimum y-coordinate Maximum y-coordinate BaseMapLayerType encapsulates the properties of a BaseMapLayer. Name of the MapLayer ResourceId of the MapLayer Whether or not the Layer can be selected Whether or not the Layer should be shown in the legend Label to be shown for the Layer in the legend Whether or not the Layer should be expanded in the legend. MapLayerGroupCommonType is a common subclass of MapLayerGroupCommonType and BaseMapLayerGroupCommonType The name of this LayerGroup Whether this group's visiblity should be visible or not when it first comes into range Whether or not the LayerGroup should be shown in the legend Whether or not the LayerGroup should be initially expanded in the legend Label to be shown for the LayerGroup in the legend BaseMapLayerGroupCommonType encapsulates the properties of a BaseMapLayerGroup. It extends MapLayerGroupCommonType by holding the layers in the group. The base map layer groups defines what layers are used to render a tile set in the HTML viewer. The layers that are part of this group. The order of the layers represents the draw order, layers first is the list are drawn over top of layers later in the list. }}} From a high-level view, the TileSetDefinition is essentially the BaseMapDefinition portion of the MapDefinition schema ripped out and put into its own resource, but with the following unique characteristics: * Ability to have its own custom tile image sizes * Ability to have its own tile image format As a forward looking measure a TileSetDefinition supports the concept of a Tile Provider (think FDO for tile sets). For the short term to see this feature realized there will only be one Tile Provider (the existing MapGuide-designated directory on the file system), but the schema provides a name/value pair mechanism for specifying connection parameters, that allows for MapGuide to support tile storage/retrieval from sources other than the MapGuide-designated file system (eg. MBTiles). The "MapGuide" tile provider has the following settings: * TileCachePath: The physical directory where rendered tiles are stored and retrieved from. This can be %MG_TILE_CACHE_PATH% to use the MapGuide default, or an aliased directory or a hard-coded physical path == APIs to support tile sets == The MgTileService will get new APIs to work with TileSetDefinitions: {{{ class MG_MAPGUIDE_API MgTileService : public MgService { PUBLISHED_API: /// Clears the entire tile cache for the given map. Tiles for all base /// map layer groups and finite scales will be removed. /// virtual void ClearCache(MgResourceIdentifier* tileSetDefinition); /// Marks a given tile cache as locked. A locked tile cache will prevent resource /// content changes in up-stream dependent resources like Layer Definitions and /// Feature Sources /// virtual void LockCache(MgResourceIdentiifer* tileSetDefinition); /// Marks a given tile cache as un-locked. An un-locked tile cache can be invalidated /// by any resource content changes in up-stream dependent resources like Layer Definitions /// and Feature Sources. /// virtual void UnlockCache(MgResourceIdentiifer* tileSetDefinition); /// Returns the default width of a tile /// virtual INT32 GetDefaultTileSizeX(MgResourceIdentiifer* tileSetDefinition); /// Returns the default height of a tile /// virtual INT32 GetDefaultTileSizeY(MgResourceIdentiifer* tileSetDefinition); /// Returns the list of available tile providers, along with supported connection parameters /// virtual MgByteReader* GetTileProviders(); }; }}} Or if we want to combine this with the HTTP 304 cacheability support {{{ class MG_MAPGUIDE_API MgTileService : public MgService { PUBLISHED_API: /// /// virtual void ClearCache(MgResourceIdentifier* tileSetDefinition); /// Marks a given tile cache as locked. A locked tile cache will prevent resource /// content changes in up-stream dependent resources like Layer Definitions and /// Feature Sources /// virtual void LockCache(MgResourceIdentiifer* tileSetDefinition); /// Marks a given tile cache as un-locked. An un-locked tile cache can be invalidated /// by any resource content changes in up-stream dependent resources like Layer Definitions /// and Feature Sources. /// virtual void UnlockCache(MgResourceIdentiifer* tileSetDefinition); /// Returns the default width of a tile /// virtual INT32 GetDefaultTileSizeX(MgResourceIdentiifer* tileSetDefinition); /// Returns the default height of a tile /// virtual INT32 GetDefaultTileSizeY(MgResourceIdentiifer* tileSetDefinition); /// Returns the list of available tile providers, along with supported connection parameters /// virtual MgByteReader* GetTileProviders(); /// Returns the timestamp of when the tile for the specified map/group/row/col/scale was generated. Returns NULL if no such tile exists /// MgDateTime* GetTileCreationDate(MgResourceIdentifier* resource, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex); /// Returns the specified base map tile for the given map. If a cached tile /// image exists it will return it, otherwise the tile is rendered and added /// to the cache. /// MgTile* GetTile(MgResourceIdentifier* resource, CREFSTRING baseMapLayerGroupName, INT32 tileColumn, INT32 tileRow, INT32 scaleIndex); }; }}} Notice that no new GetTile overload is introduced (assuming this feature is separate from the HTTP 304 cacheability support). This is because tile access requires the same set of parameters, except that instead of a Map Definition, we can specify a TileSetDefinition instead. On the implementation side, the GetTile methods need to support TileSetDefinition as a valid resource type and be able to resolve the physical tile storage location to fetch and store tiles in from the TileSetDefinition. For a first cut implementation, it is safe to just hard-code this logic to resolve to the same physical tile cache directory that the existing MapGuide tile cache code uses. The GetTileProviders method is akin to GetFeatureProviders in MgFeatureService. It is a method to assist client-side authoring tools to provide rich user interfaces for editing TileSet Definitions. A sample response looks like this: {{{ Default Default Tile Provider Default tile access provided by MapGuide. Supports MapGuide-managed tile path or user-defined path TilePath TilePath %MG_TILE_CACHE_PATH% }}} == Tile Set Locking == To prevent accidental invalidation due to editing upstream dependent resources, the MgTileService will support the concept of locking and unlocking TileSetDefinition resources. When a TileSetDefinition is saved for the first time as a new resource it is in a default unlocked state. An unlocked Tile Set is susceptible to all the usual tile cache invalidation mechanisms: * The TileSetDefinition is overwritten * An upstream Layer Definition or Feature Source has changed When a TileSetDefinition is locked with the MgTileService::LockCache() API, the above scenarios will be prevented via throwing the appropriate MgException (class type TBD) on saving of resource content on a TileSetDefinition or any of its upstream resource types. The exception thrown should carry the resource id of the downstream TileSetDefinition that would be affected by this operation so it can be relayed to the user or client application that they can't save this resource because it would invalidate a locked tile set. The act of locking a TileSetDefinition is basically saying that this tile cache and all upstream resources are "read-only" and its tile cache can not be invalidated unless explicitly unlocked. The act of having a TileSetDefinition unlocked is saying that this tile cache could be invalidated at any time. The actual implementation details are still TBD. The tentative idea would be to leverage the existing server Resource Service APIs (ie. DBXML) to store the tile set lock information, relieving us of synchronization/multi-threaded concerns to read/write lock state (Further discussion required on whether this is a good idea) == Map Definition Schema changes == To retain the additive qualities of previous schema revisions. The Map Definition schema will simply support a new optional TileSetSource element which is simply a reference to a TileSetDefinition {{{ ... A reference to the tile set source to use ResourceId of the TileSetDefinition }}} Only up to 1 TileSetSource can be defined for any Map Definition as having multiple TileSetSources presents complex situations like how to address disparity of scale ranges between different TileSetSources. Because TileSetDefinitions are bound to a specific coordinate system, the CoordinateSystem element of the Map Definition will no longer be the single point of truth when determining if dynamic layers in a map require re-projection. Instead if a Map Definition references a TileSetDefinition, its CoordinateSystem element takes precedence over the one defined in the Map Definition. If a TileSetDefinition is referenced, all dynamic layers in the Map Definition will re-project to that Coordinate System instead of the one in the Map Definition. This requirement can be relaxed/lifted if/when we support raster re-projection of existing rendered tiles (Discussion required: Is this feasible?) In terms of parsing into a MgMap at runtime, the TileSetSource and BaseMapDefinition are mutually exclusive. In the event that both are defined (because the XML schema allows for it), the BaseMapDefinition element "wins". === MgMap changes === Because Map Definitions are parsed into MgMap objects at runtime, we need to update MgMap and its associated classes (MgLayer/MgLayerGroup) to also take into account support for TileSetDefinitions The GetMapSRS() method will stay the same, but its behaviour will change. It will now return the CoordinateSystem of the referenced TileSetDefinition if one exists. Depending on which one is defined (or which one "wins" under our rules of element precendence), the finite scale list will be initialized from either the BaseMapDefinition element or the referenced TileSetDefinition The MgLayerGroupType class will introduce a new constant {{{ class MgLayerGroupType { PUBLISHED_API: /// Specifies that the layer is a base map layer from a TileSetDefinition resource /// static const INT32 BaseMapFromTileSet = 3; } }}} The MgLayerGroup will have a new API to return the TileSetDefinition resource id where this group comes from {{{ class MG_PLATFORMBASE_API MgLayerGroup : public MgNamedSerializable { PUBLISHED_API: /// Returns the resource id of the Tile Set Definition that this group originates from. /// Returns NULL if GetLayerGroupType() is not MgLayerGroupType::BaseMapFromTileSet /// MgResourceIdentifier* GetTileSetDefinition(); } }}} == Client application logic changes to support TileSetDefinition resources == The beautiful thing about this new feature is that from the client application perspective, the calling interface remains the same. The only difference is that the Map Definition can now also be a TileSetDefintion, and client applications should know which resource type to use with the new API in MgLayerGroup = WMS = [ticket:286 WMS Cache headers] - see above. [ticket:285 Publish MapDefinitions via WMS] Currently only layers are exposed via WMS, maps could be as well. The WMS service doesn't currently use the tile cache, which means every WMS request is rendered. [[BR]] Linking the WMS service up to utilize the tile cache would dramatically improve the performance [[BR]] and capacity of WMS with Mapguide.[[BR]] = TMS = [wiki:GoogleSoC2007#TileMapServicePublishing TMS]