MapGuide RFC 2 - Unmanaged Data APIs

This page contains an change request (RFC) for the MapGuide Open Source project. More MapGuide RFCs can be found on the RFCs page.

Status

RFC Template Version	(1.0)
Submission Date	1 November 2006
Last Modified	Tony Fang Timestamp
Author	Tony Fang
RFC Status	adopted
Implementation Status	under development
Proposed Milestone	1.2
Assigned PSC guide(s)	(when determined)
Voting History	Dec 18, 2006
+1	Tom, Jason, Andy, Bruce, Haris, Paul, Bob
+0
-0
-1

Overview

Managed data is data contained inside the server's data repository. Unmanaged data is any data outside of the server's data repository.

We want to add server/web functionality and APIs which will allow Studio to create feature sources from unmanaged data. To accomplish this, directories accessible by the server machine can be specified as unmanaged data directories. APIs will be added which will enumerate the available unmanaged data files from the specified unmanaged data directories.

WebStudio can also make use of these APIs to create unmanaged feature sources.

Motivation

When Studio is used to create a feature source (e.g. using Load Procedure), the data files (e.g. sdf, shp, raster, dwf, odbc) are uploaded to the server's repository (managed data). This is not an issue when the data files are small. If Studio and Server are on the same machine, this uploading is an unnecessary duplication of the data file. If the data file is really large, the copying is slow and it consumes unnecessary hard-drive space.

Currently, a feature source which references unmanaged data can be created. But it must be done manually. The exact path to the data file on the server machine must be known. Then the Feature Source XML must be manually edited and uploaded to the server via the web tier. This is a cumbersome process.

We want to automate this workaround in Studio. To do this, additional APIs and functionality will be added to the server/web. Directories accessible by the server machine can be specified as unmanaged data directories. APIs will be added which will enumerate the available unmanaged data files. Thus direct access to the server machine's file system will no longer be required. Creation of unmanaged data feature sources can now be created remotely.

The API will only enumerate data files contained in the directories and subdirectories specified by the server. For security reasons we do not want to allow the API to return data files in directories which are not specified. For windows, specified directories can be physical drives or network paths. For linux, it can be any accessible path.

Proposed Solution

The server's unmanaged data directories will be specified using the serverconfig.ini. A new section defining directory mappings will be used to specify the unmanaged data directories. This section can be edited using the Server Admin API.

For windows, it may look like this:

[UnmanagedDataMappings]
SomeSdfFiles = c:\mydata\sdf
Some Shp Files = d:\otherdata\shp
BigArseImages = \\some_other_machine\data\images
Some 很大中文 DwfFiles = c:\mydata\很大中文Dwf
大きい　ｓｄｆ = c:\mydata\bigsdf

For linux, it may look like this:

[UnmanagedDataMappings]
SomeSdfFiles = /usr/mydata/sdf
Some Shp Files = /usr/otherdata/shp
BigArseImages = /mnt/some_other_machine/data/images
Some 很大中文 DwfFiles = /usr/mydata/很大中文Dwf
大きい　ｓｄｆ = /usr/mydata/bigsdf

Unicode characters are supported in the mapping names. Spaces in the mappings are also supported. Square brackets [ and ] are not allowed. They are reserved characters for the serverconfig.ini section titles.

We will add API methods to return the unmanaged data mappings, set the mappings, and also verify that the mappings are valid and accessible.

We will add an API to enumerate the unmanaged data files available on the server machine. The files will be prefaced with their mapping names. You may enumerate all files from all the available drive mappings, or you may select a single drive mapping.

A returned list from all available drive mappings may look like this:

[SomeSdfFiles]world.sdf
[Some Shp Files]ecuador.shp
[Some 很大中文 DwfFiles]large.dwf
[Some 很大中文 DwfFiles]subdir/Big.dwf
[大きい　ｓｄｆ]reallybig.sdf

The server will not be locked down such that it is restricted to those files in the Unmanaged Data Mappings. In other words, this means that a feature source can still reference a file that is not in one of the subdirectories specified by the mappings. The reason we don't have to restrict to those subdirectories is because we can use the operating system's or network's built-in security to limit access to files. It will be dependent upon the administrator to allow access to the files in the Unmanaged Data Mappings, otherwise they won't be very useful. Thus manually created unmanaged features sources and those created by webstudio will continue to work.

For cross platform compatibilities (i.e. Linux only supports ‘/’s while Windows supports both ‘/’s and ‘\’s), pathnames referenced in XML documents should always contain forward slashes (‘/’s) instead of backward slashes (‘\’s).

Technical

The following API functions will be added to the Resource Service:

virtual MgStringCollection * MgResourceService::EnumerateUnmanagedData(CREFSTRING mappingName, CREFSTRING dataTypeFilter, INT32 depth);

Enumerates the available unmanaged data from the server.

Parameters:

mappingName specifies which mapping to use to return the unmanaged data. When set to empty string, it will use all mappings.

dataTypeFilter is a filter on the returned data. E.g. SHP, SDF, DWF, RASTER, FOLDER. When set to empty string, it will return all data types. FOLDER returns a list of subfolders.

depth is how many levels of subfolders it should search for data. -1 to search all depths, 0 to search only the top level, 1 to search 1 level, 2 to search 2 levels, etc.

Returns:

Returns an MgStringCollection containing the available unmanaged data files.


virtual MgStringCollection * MgResourceService::EnumerateUnmanagedDataMappings();

Enumerates the unmanaged data mappings.

Parameters:

None.

Returns:

Returns an MgStringCollection containing only the mapping names. 

Implications

API documentation will need to be updated.

Test Plan

• Add webtier test pages which use the new API methods.
• Add unit tests which test the new API methods.

Funding/Resources

Autodesk to supply the resources to make the required changes.