Node Import

Description

Node import is a flexible, high-performance service for loading data into Advisor.

Together with the related Node Export service it can be used in many different ways:

  • To backup and restore data.
  • To export data to other systems and load data from other systems.
  • To integrate with file-based processing.
  • To perform copies of entire structures across Advisor.
  • To synchronise data between instances of Advisor.

Load set

The nodes to be loaded are referred to as the "load set".

In the unusual circumstance that the load set has missing intermediate packages, the intermediate packages are also added to the load set.

Validation

To preserve performance, the import process assumes as far as possible that the import data is correct, and only checks what it has to check to enforce the integrity and authority structure for the system. Specifically, it checks that:

  • The user is authorised to administer nodes that may be overwritten.
  • That the user is authorised to link to any referenced targets outside the node set.
  • That the user is authorised to use any referenced node type outside the node set.
  • That the user is authorised to use packages inside which nodes are created.

To allow data to be loaded quickly, it does not perform normal validation checks on the data in the load set, including:

  • Member types on node types.
  • The storage model of individual member types, i.e. it loads whatever value, scale and target is passed.
  • Member cardinality.
  • Link and use type permissions within the load set, as the user is presumed to have all of these.

The load does not run any validation or trigger scripts.

Provided that the user is authorised to link to them, targets are not checked against target lists. However, target versions are modified to match those listed in the target list of the member type.

Permissions

The importing user must have:

  • admin permissions on any node that is overwritten or merged.
  • use package or equivalent permission (such as admin) on any packages inside which nodes are created.
  • use type permission on any node outside the load set that are used as node types in the node set.
  • link permissions on any nodes outside the node set that are the targets of links in the node set.

The importing user is granted:

  • admin permission on any nodes that are created
  • any additional use type permission on node types within the node set

The import does not grant any other pemissions.

Identifying change

The import only actually updates nodes and members where the data has changed. You can therefore apply a full node export to existing structures, and only changes will be applied. This reduces the need to re-derivation of the nodes, and prevents new versions being created unnecessarily.

Commit processing

The import performs all validation before starting the load, so the load either works completely or not at all.

The service commits after each node is imported in order to free up database resources. However, the loaded nodes should not be accessed until the load is complete because the nodes on which they depend may not be correctly populated.

It is theoretically possible to disrupt the node by deleting nodes on which the load depends as the load take place. In this case, the load will raise a database error and stop, though the data should be in a clean state.

Input

<NodeImport>
<userLogonReference>userid</userLogonReference>
<password>password</password> <action>insert|overwrite|merge|synchronize</action> <importVersions>preserve|latest|current|best</import> <createPackages>false|true</createPackages> <createTargets>false|true</createTargets> <createTypes>false|true</createTypes> <check>false|true</check> <Node> <type/> <reference/> <delete>false|true</delete> <Member> <type/> <value/> <scale/> <target/> </Member> </Node> <nodeVersionReferenceFile/> <importFrom/> <importTo/> </NodeImport>

action specifies what the import should do if the imported nodes already exist:

  • insert - the import should only create new nodes. If any of the nodes already exist, raise a validation error. This is the default.
  • overwrite - the import can overwrite existing nodes. If the nodes are in draft, they are completely replaced with the new data. If they are frozen, new versions are created.
  • merge - the import can overwrite existing nodes, but only the members types that are passed in the import are replaced. This is useful for partial loads, for example loading a revised set of descriptions but not modifying the names.
  • synchronize - like overwrite, except that all nodes within the importTo package that are not in the load set are also deleted. importTo must be set.

    synchronize does not synchronize the importTo package itself (only the nodes in the package). The importTo package is updated if data for it is passed, but not deleted if not passed.

    Be very cautious using synchronize. If, for example, it deletes node types, this can cause a cascade of delete of node instances. If in the source you have renamed a node type, you could find all the instances are deleted when you synchronize. Always perform a check on a synchronize. Consider freezing packages before synchronization.

importVersions specifies which versions should be used for node types and targets outside of the load set. It has not effect on nodes within the load set, which always use the latest version.

  • preserve - use the version numbers from the request. If the request does not have version numbers, use the latest version. This is the default.
  • latest - ignore the version numbers on the request, always use the latest version.
  • current - ignore the version numbers on the request, always use the current version.
  • best - ignore the version numbers on the request. Use the current version is there is one, or else use a draft version.

check specifies whether the load process should stop after is has completed validation, and should not continue with the load even if there are no errors. The default is false.

createPackages specifies whether, when new nodes are created, missing packages should also be created as null nodes. Default is false.

createTargets specifies whether targets referenced in the load but which do not exists and are not in the load set should be created. Default is false. If true, then any missing packages for those targets are also created. Set this to true to enable a load to be performed in parts.

createTypes specifies whether node types and member types referenced in the load but which do not exist and are not in the load set should be created. Default is false. If true, then any missing packages for those node types or targets are also created. Set to true to enable a load to be performed in parts.

check specifies whether the import should stop after resolving the import, and just show what it would do. It output the options and data (elaborated with version numbers and null nodes, and with nodes and members that will not be updated removed). All node deletions in the import are shown, whether or not the nodes already exist.

The Node and Member specify the data to be loaded. Node repeats, and Member repeats within node.

Node type is optional. If it is omitted:

  • For newly created nodes, a null type is used.
  • For existing nodes, the node type is not changed.
  • For new versions, the node type of the previous version is used.

The local part of the reference may start with "$auto", in which case a unique node reference will be created. Anything after the $auto (excluding an optional starting _) will be used to build the local reference, but must be a valid local reference. For example "$auto_PeterRabbit" will attempt to use the local reference PeterRabbit, but, if that is in use, generate a new local reference based on PeterRabbit. These references can be used as targets from other nodes.

If delete is set to true, the node will be deleted. delete defaults to false. Node deletion is complicated:

  • Node deletion takes place after other nodes and node versions have been created and updated. You can think of a delete as a "after the import, delete this". Deleting a package can result in the deletion of newly-created nodes.
  • The deleted node may be referenced by other nodes in the import.
  • Nodes or node versions may be created and then deleted to correctly process dependencies on other nodes. If you have specified that you want to delete a node that does not exist, you must be authorised to create the node.
  • If you delete a non-existent node in a non-existent package, the non-existent package may be created. You should not rely on this behaviour.
  • Frozen node versions are not deleted. This is not an error.
  • When the check option is set, node deletions are always shown on the output even if the node does not exist or is not currently in draft.
  • Node deletion is much slower than other parts of the import.

When action is merge, a member with no value, no scale and no target can be used to delete all the members for a member type. If you use this to delete all members from a member type, only pass one empty member.

nodeVersionReferenceFile specifies a file node from the related file of which input nodes and options should be read. These are read before those in the body of the request, i.e. the request overrides the ones from the file. The file is assumed to be in XML format, unless it has an extension of "zip", in which case it is assumed to be in zipped XML format, with the data in entry "node.xml". The (uncompressed) file follows has the same format as the output from NodeExport or the input to NodeImport.

<NodeImport>
<Node> <type/> <reference/> <Member> <type/> <value/> <scale/> <target/> </Member> </Node> </NodeImport>

It can use any root element name. All elements other than node elements are ignored.

The following options can be read from the input file or the request document: action, importVersions, createPackages, createTargets, createTypes, importFrom and importTo.

Within the import file, for members of type system.FILE_STORAGE_KEY, a prefix of "$base64:" indicates that the remaining value is the base64-encoded version of the file. A prefix of "$content:" indicates that the remaining value is an unencoded version of the file.

importFrom and importTo specify package references. Any nodes in the load set that are in package importFrom should be loaded into package importTo. Both importFrom and importTo must be present for this to work. importFrom and importTo should not contain version numbers.

Output

<NodeImport>
<errorNumber>0</errorNumber>
<importTo/>
</NodeImport>

If there are any validation errors, the service will return a user error.

comment and timestamp are returned if present in the input data, but are not otherwise returned or validated.

importTo is only returned if importTo is passed in, and contains the node reference of the importTo value. This can be useful when used with an importTo with a $auto local reference.

If the check options is set to true, or if there are any errors, the service will return the Node and Member structures that are in error or, if there are no errors and the check option is true, the structures that will be applied. The format is the same as the Node and Member structures on import, except that the Node and Member may also have an error field to indicate errors.

If the check option is true, or if there are errors, the service will also return the action, import versions, create packages, create targets and create types options. (It does not return the import from and import to options because these are resolved into the data).

Errors

103 - Parameter error

105 - User error

Class

com.metrici.xerula.NodeImportService