Metrici holds two sets of member data:
- Original data, which is data entered into the system.
- Derived data, which is calculated.
Write operations write to original members, and read operations read from derived members.
To make sure that the data matches, whenever a node created or changed it is set as out-of-date. If an out-of-date is read, a derivation process runs to calculate the derived data. In this way, nodes are always kept up-to-date.
The setting out of date and derivation processes take account of linked nodes. When a node is changed, then all the nodes that link to it are also set out-of-date. Before a node is derived, all the out-of-date nodes it links to are derived.
By default, the derivation process deletes all the derived members and then copies all the original members to become derived members. It checks the derived data to pick up any changes made to the Name (system.NODE_NAME) member.
The derivation process can be modified by scripts defined on the node's node type and member types. See Derivation scripts for more details. Member types are used extensively to create calculated data.
Inheritance is a special form of derivation, in which the derived data of one node includes all the derived data of another node.
Freshness
By default, out-of-date nodes are re-derived before they are read. The tag Allow Stale Read (system.ALLOW_STALE_READ_TAG) can be added to a node to allow the user interface to retrieve a stale node without performing derivation. This is particularly useful on packages that are constantly set out of date because of changes to their contained nodes.
Derivation can be forced by adding refresh=true to the URL, or by using the Recalculate option from the More menu. This is used to pick up changes that have not been automatically applied.
A node can be forced out-of-date and then re-derivedy adding refresh=full to the URL, or by using the Full recalculate option from the More menu. This can be useful to re-trigger failed derivation in dependent nodes.
Node access in the scripting API supports the concept of freshness, which can be used to retrieve stale nodes, refresh a node, or force recalculation.
Update behaviour
When a node changes, its containing packages is not by default set out of date. This behaviour can be modified by setting the Package Content Change Behaviour (system.PACKAGE_CONTENT_CHANGE_BEHAVIOUR) of the package's node type. See Standard metadata for more details.
When a node changes, all the nodes that link to it are by default set out of date. This behaviour can be modified by adding the Ignore Target Change (system.IGNORE_TARGET_CHANGE_TAG) tag to the member type, which prevents out-of-date conditions being propaged through links using that member type.
Background derivation
Derivation is triggered whenever an out-of-date node is accessed. In addition, a background process identifies out-of-date nodes and performs derivation on them. The background derivation process maintains statistics and applies a scheduling algorithm to ensure that priority is given to quick recalculations, that nodes are typically derived before the nodes that depend on them, and that long-running derivations are not scheduled to run very often.
Delayed derivation
It can in some cases be useful to schedule how often derivation should occur, or to trigger long-running processes when nodes change.
The delayed derivation components support this. A delayed derivation node indicates one or more nodes that it depends on. When the nodes it depends on change, background derivation will trigger the delayed derivation component, which in turn schedules a service (using Schedule Service) to execute the derivation component to perform additional processing. The scheduling is parameterised by two numbers, the calculation delay and the calculation period. After a change is detected, the processing will be scheduled to run after calculation delay. If multiple changes are detected, the procesings will run at most every calculation period. Unlike running on a fixed schedule, if no changes are made, the processing is not run.
A typical use of delayed derivation is to run reporting processes, for example to create occasional snapshots of incoming survey data. Delayed derivation allows the recalculation of the snapshot to be limited to, say, once every two hours, rather than triggering it every time a survey is submitted.
Delayed derivation components provide an option for the user to run the processes straight away, either in the foreground attached to the browser session, or scheduled in the background as soon as possible. In the reporting scenario above, they provide the user with an "update now" option to bring the snapshot up to date, in addition to the periodic automated update.
The main derivation component is Delayed Derivation (library.parts.DelayedDerivationType). This documents the parameters and inputs in detail. There are other specialised types:
- Delayed Recalculation (library.parts.DelayedRecalculationType) schedules a full recalculation on one or more nodes.
- Permission Manager Updater (library.permission.PermissionManagerUpdaterType) is used in conjunction with Permission Manager (library.permission.PermissionManagerType) to apply role-based permissions to a package structure and maintain them automatically.