= !MapGuide RFC 12 - Unmanaged Data API Changes =

This page contains an change request (RFC) for the !MapGuide Open Source project.  
More !MapGuide RFCs can be found on the [wiki:MapGuideRfcs RFCs] page.


== Status ==
 
||RFC Template Version||1.0|| 
||Submission Date||15 January 2007||
||Last Modified||jasonbirch [[Timestamp]]||
||Author||Tony Fang||
||RFC Status||adopted||
||Implementation Status||completed||
||Proposed Milestone||1.2||
||Assigned PSC guide(s)||Tom Fukushima||
||'''Voting History'''||Jan 24, 2007||
||+1||Tom, Bruce, Jason, Paul, Haris, Andy, Bob||
||+0||||
||-0||||
||-1||||
 
== Overview ==

Changes to the Unmanaged Data API specified in http://wiki.osgeo.org/index.php/MapGuide_RFC_2_-_Unmanaged_Data_APIs. 

Also changes to the Server Admin PHP pages allowing editing of the serverconfig.ini are specified in more detail.

== Motivation ==

These changes will: 
 * allow scaling to sites containing many unmanaged data resource files.
 * disambiguate the response.
 * treat the alias component as a folder grammatically.
 * simplify the API implementation.
 * eliminate the need for the redundant !EnumerateUnmanagedDataMappings API method.
 * give the client better control of desired query results.

== Proposed Solution ==

These are the proposed changes to the Unmanaged Data APIs.
 * Remove !EnumerateUnmanagedDataMappings method.
 * Modify !EnumerateUnmanagedData API method as specified in the Technical section.
   * modify arguments to ''path'', ''recursive'', ''select'', and ''filter''
   * ''path''="", ''recursive''=false, ''select''="FOLDERS" yields the same result as !EnumerateUnmanagedDataMappings
   * All file types are returnable with a client-customizable filter
   * Return an !UnmanagedDataList (see !UnmanagedDataList-1.0.0.xsd below) instead of a !StringCollection.
 * Windows hidden folders and files are returned (already works this way after RFC2 implementation).
 * All other aspects of this API method remain the same as before.
   * Unicode mappings and filenames are supported.
   * Square brackets are not allowed for mapping names.
   * Spaces are allowed in mapping names.
   * Mappings can be physical drives or network paths.

These additions to the Server Admin PHP pages will allow editing of the !UnmanagedDataMappings section in the serverconfig.ini.

 * Below "Configure WFS", add an entry called "Configure Unmanaged Data".
 * "Configure Unmanaged Data" will display a table with 3 columns: 
   * radio button denoting current selection
   * Mapping Name
   * Location
 * The table will also have 3 buttons:
   * "Add Mapping" will display a page which allows you to add a new mapping.
   * "Edit Mapping" will display a page which allows you to edit the currently selected mapping location.
   * "Delete Mapping" will delete the currently selected mapping.
 * This is similar to the "Manage Users" table, except with only 3 columns.
 * The server must be re-started before any Unmanaged Data serverconfig.ini changes will take effect.

[[Image(AddUnmanagedData.jpg, 400)]][[BR]]

[[Image(EditUnmanagedData.jpg, 400)]][[BR]]

[[Image(ConfigureUnmanagedData.jpg, 400)]][[BR]]



=== Technical ===

virtual !MgByteReader  * MgResourceService::!EnumerateUnmanagedData(CREFSTRING ''path'', bool ''recursive'', CREFSTRING ''select'', CREFSTRING ''filter'');

Enumerates the available unmanaged data from the server.

'''Parameters:'''

''path'' Specifies an unmanaged data mapping with optional subfolder path. It must begin with a legitimate mapping. e.g. "[alias1]" or "[alias1]subfolder1/subfolder2". It cannot begin with a drive or folder path. Default value is "", which corresponds to the "Unmanaged Data Root". At the top level of the "Unmanaged Data Root", you can only see the unmanaged data mappings. No data files can be seen in the top level of the "Unmanaged Data Root".

''recursive'' If it is true, it will recurse into all subfolders looking for files and/or subfolders. If it is false, it will only seek files and/or subfolders at the current level. Default value is false.

''select'' Specifies the type of data to return. The choices are "FILES", "FOLDERS", or "BOTH". Default value is "FOLDERS".

''filter'' Specifies an optional file extension filter on files returned. Use a ";" to delimit multiple file extension filters. e.g. "shp", "shp;shx;prj", or "jpg;jpeg;jpe;bmp;png". The file filter is case-insensitive.
If ''select''="FOLDERS", then ''filter'' is ignored. Default value is "", which returns all files.


'''Returns:'''

Returns an !MgByteReader object containing the list of all unmanaged data files and/or folders. The list is in XML format using the !UnmanagedDataList  schema.

The new schema will be named ''!UnmanagedDataList-1.0.0.xsd''. This is closely analogous to ''!ResourceList-1.0.0.xsd'' which specifies the response of !EnumerateResources.

{{{
  <xs:element name="UnmanagedDataList">
    <xs:annotation>
      <xs:documentation>Unmanaged data list</xs:documentation>
    </xs:annotation>
    <xs:complexType>
      <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="UnmanagedDataFolder">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="UnmanagedDataId" type="xs:string"/>
              <xs:element name="CreatedDate" type="xs:dateTime"/> (MAYBE)
              <xs:element name="ModifiedDate" type="xs:dateTime"/> (MAYBE)
              <xs:element name="NumberOfFolders" type="xs:integer"/>
              <xs:element name="NumberOfFiles" type="xs:integer"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="UnmanagedDataFile">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="UnmanagedDataId" type="xs:string"/>
              <xs:element name="CreatedDate" type="xs:dateTime"/> (MAYBE)
              <xs:element name="ModifiedDate" type="xs:dateTime"/> (MAYBE)
              <xs:element name="Size" type="xs:integer"/> (MAYBE)
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:choice>
    </xs:complexType>
  </xs:element>
}}}

=== API Examples ===

==== Windows Server ====

serverconfig.ini contains:

{{{
[UnmanagedDataMappings]
alias1 = c:\mydata\sdf
alias2 = \\other_machine\public_data
}}}

The server machine contains the following files:

 * c:\mydata\sdf\sewers.sdf
 * c:\mydata\sdf\buildings.sdf
 * c:\mydata\sdf\canada\bc.sdf
 * c:\mydata\sdf\canada\ab.sdf
 * c:\mydata\sdf\canada\sk.sdf
 * \\other_machine\public_data\readme.txt
 * \\other_machine\public_data\raster\world.bmp
 * \\other_machine\public_data\raster\africa.tiff
 * \\other_machine\public_data\raster\usa\usa.jpeg
 * \\other_machine\public_data\shp\squares.shp
 * \\other_machine\public_data\shp\circles.shp


'''!EnumerateUnmanagedData API Examples on Windows server'''

||''path''||''recursive''||''select''||''filter''||FOLDERS RESULT||FILES RESULT||Notes||
|| ""|| false|| "FOLDERS"|| ""|| [alias1][[BR]][alias2]|||| Equivalent to removed !EnumerateUnmanagedDataMappings. This is the "Unmanaged Data Root." ||
|| ""|| true|| "FOLDERS"|| ""|| [alias1][[BR]][alias1]canada/[[BR]][alias2][[BR]][alias2]raster/[[BR]][alias2]raster/usa/|||| Note the forward slash. ||
|| ""|| false|| "FILES"|| ""|||||| No files returned. ||
|| ""|| true|| "FILES"|| ""|||| [alias1]sewers.sdf[[BR]][alias1]buildings.sdf[[BR]][alias1]canada/bc.sdf[[BR]][alias1]canada/ab.sdf[[BR]][alias1]canada/sk.sdf[[BR]][alias2]readme.txt[[BR]][alias2]raster/world.bmp[[BR]][alias2]raster/africa.tiff[[BR]][alias2]raster/usa/usa.jpeg[[BR]][alias2]shp/squares.shp[[BR]][alias2]shp/circles.shp|| All files returned. ||
|| ""|| true|| "BOTH"|| ""|| [alias1][[BR]][alias1]canada/[[BR]][alias2][[BR]][alias2]raster/[[BR]][alias2]raster/usa/|| [alias1]sewers.sdf[[BR]][alias1]buildings.sdf[[BR]][alias1]canada/bc.sdf[[BR]][alias1]canada/ab.sdf[[BR]][alias1]canada/sk.sdf[[BR]][alias2]readme.txt[[BR]][alias2]raster/world.bmp[[BR]][alias2]raster/africa.tiff[[BR]][alias2]raster/usa/usa.jpeg[[BR]][alias2]shp/squares.shp[[BR]][alias2]shp/circles.shp|| All folders and files returned. ||
|| "[alias1]"|| false|| "FILES"|| ""|||| [alias1]sewers.sdf[[BR]][alias1]buildings.sdf[[BR]]||||
|| "[alias2]raster/usa"|| true or false|| "FILES"|| ""|||| [alias2]raster/usa/usa.jpeg[[BR]]|| The result is the same whether recursive is true or false. ||
|| "[alias2]" or ""|| true|| "FILES"|| "bmp;tiff"|||| [alias2]raster/world.bmp[[BR]][alias2]raster/africa.tiff[[BR]]|| The result is the same whether path is "[alias2]" or "". ||

==== Linux Server ====

serverconfig.ini contains:

{{{
[UnmanagedDataMappings]
alias1 = /user/Thelonious/sdf
alias2 = /mnt/other_machine/public_data
}}}

The server machine contains the following files:

 * /user/Thelonious/sdf/'''sewers.sdf'''
 * /user/Thelonious/sdf/'''Sewers.SDF'''
 * /user/Thelonious/sdf/readme.txt
 * /user/Thelonious/sdf/sewers.sdf.bak
 * /mnt/other_machine/public_data/readme.txt
 * /mnt/other_machine/public_data/raster/world.bmp
 * /mnt/other_machine/public_data/raster/africa.tiff
 * /mnt/other_machine/public_data/raster/usa/usa.jpeg
 * /mnt/other_machine/public_data/shp/squares.shp
 * /mnt/other_machine/public_data/shp/circles.shp

'''!EnumerateUnmanagedData API Examples on Linux server'''

||''path''||''recursive''||''select''||''filter''||FOLDERS RESULT||FILES RESULT||Notes
|| "" || false || "FOLDERS" || "" || [alias1][[BR]][alias2] || || Equivalent to removed !EnumerateUnmanagedDataMappings. This is the "Unmanaged Data Root." ||
|| "" || true || "FOLDERS" || "" || [alias1][[BR]][alias2][[BR]][alias2]raster/[[BR]][alias2]raster/usa/ || || Note the forward slash. ||
|| "" || false || "FILES" || "" || || || No files returned. ||
|| "" || true || "FILES" || "" || || [alias1]sewers.sdf[[BR]] [alias1]Sewers.SDF[[BR]][alias1]readme.txt[[BR]][alias1]sewers.sdf.bak[[BR]][alias2]readme.txt[[BR]][alias2]raster/world.bmp[[BR]][alias2]raster/africa.tiff[[BR]][alias2]raster/usa/usa.jpeg[[BR]][alias2]shp/squares.shp[[BR]][alias2]shp/circles.shp || All files returned. Notice sewers.sdf and Sewers.SDF are both returned. ||
|| "" || true || "BOTH" || "" || [alias1][[BR]][alias2][[BR]][alias2]raster/[[BR]][alias2]raster/usa/[[BR]]|| [alias1]sewers.sdf[[BR]][alias1]Sewers.SDF[[BR]][alias1]readme.txt[[BR]][alias1]sewers.sdf.bak[[BR]][alias2]readme.txt[[BR]][alias2]raster/world.bmp[[BR]][alias2]raster/africa.tiff[[BR]][alias2]raster/usa/usa.jpeg[[BR]][alias2]shp/squares.shp[[BR]][alias2]shp/circles.shp || All folders and files returned. Notice sewers.sdf and Sewers.SDF are both returned. ||
|| "[alias1]" || false || "FILES" || "sdf" || || [alias1]sewers.sdf[[BR]][alias1]Sewers.SDF || Note the mixed case of the sdf extension. Notice sewers.sdf and Sewers.SDF are both returned. ||
|| "[alias2]raster/usa" || true or false || "FILES" || "" || || [alias2]raster/usa/usa.jpeg[[BR]] || The result is the same whether recursive is true or false. ||
|| "[alias2]" or "" || true || "FILES" || "bmp;tiff" || || [alias2]raster/world.bmp[[BR]][alias2]raster/africa.tiff[[BR]] || The result is the same whether path is "[alias2]" or "". ||


== Implications ==

API documentation will need to be updated.

== Test Plan ==

 * Update webtier test pages to use the updated API methods.
 * Remove test for eliminated !EnumerateUnmanagedDataMappings.

== Funding/Resources ==

Autodesk to supply the resources to make the required changes.

== Addendum ==

For a feature source, an alias must be in the form:
%MG_DATA_PATH_ALIAS[alias]%

This follows the existing conventions in the server for replacing text tags (e.g. %MG_DATA_FILE_PATH%). The server performance will be much better compared to using only [alias] as it will reduce the amount of potential false-positive hits while performing the alias re-mapping.

Example:
{{{
<?xml version="1.0" encoding="utf-8"?>
<FeatureSource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xsi:noNamespaceSchemaLocation="FeatureSource-1.0.0.xsd">
  <Provider>OSGeo.SDF</Provider>
  <Parameter>
    <Name>File</Name>
    <Value>%MG_DATA_PATH_ALIAS[alias1]%LIPOFGUTTER.sdf</Value>
  </Parameter>
  <Parameter>
    <Name>ReadOnly</Name>
    <Value>TRUE</Value>
  </Parameter>
</FeatureSource>
}}}

 
For the !EnumerateUnamangedData API method, the "select" parameter should be renamed to "type". This follows the convention used by the other API methods.

{{{
virtual !MgByteReader  * MgResourceService::!EnumerateUnmanagedData(CREFSTRING ''path'', bool ''recursive'', CREFSTRING ''type'', CREFSTRING ''filter''); 
}}}


Here is the http signature for the !EnumerateUnmanagedData API:
{{{
OPERATION=ENUMERATEUNMANAGEDDATA
VERSION=1.0.0
PATH=
RECURSIVE=0
TYPE=Folders
FILTER=
}}}