Data hub configuration

Insight Hub requires a configuration of the data hub. This may need to be modified for customer needs, but this section describes a model setup which may be useful as a basis. More details are available in the Data Hub configuration.

Database setup

The Insight Hub release includes a ddl.sql file which can be used to create a MariaDB database to support Insight Hub. The notes below presume this is set up using a database name of "insight".

Web application setup

/var
  /www
    /datahub
      /WEB-INF
        web.xml
        /lib
          datahub-dist-V.v.jar

web.xml is very simple. It maps all incoming requests to the Data Hub instance servlet, and loads a Monitor servlet to perform background processing.

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">

  <context-param>
    <param-name>propertyDir</param-name>
    <param-value>/var/datahub/config</param-value>
  </context-param>
  
  <servlet>
    <display-name>Monitor</display-name>
    <servlet-name>Monitor</servlet-name>    
    <servlet-class>com.metrici.datahub.MonitorServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
  </servlet>  
  
  <servlet>
    <display-name>Instance</display-name>
    <servlet-name>Instance</servlet-name>
    <servlet-class>com.metrici.datahub.InstanceServlet</servlet-class>
  </servlet>

  <servlet-mapping>
    <servlet-name>Instance</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
 
</web-app>

If you want to put the datahub directories somewhere other than /var/datahub/config, set the propertyDir context parameter to the appropriate directory.

datahub-dist-V.v.jar is the Data Hub distribution jar, where V.v is the major/minor version.

server.xml

Within /etc/tomcat9/server.xml, add a <Context> element within the <Host> element.

<Context docBase="/var/www/datahub" path="/datahub"> 
</Context>

Datahub configuration

/var
  /datahub
    /config
      datahub.properties
      /insight
        authority.json
        datahub.properties
        public_key.pem
        /schema
          default.json
          load.json
          search.json
    /filestore
      /insight
    /filestorePurge

There are two datahub.properties file, a top-level one at /var/datahub/config/datahub.properties, and an insight-specific one at /var/datahub/config/insight/datahub.properties.

The top-level datahub.properties provides properties for the instance servlet and monitor, plus defaults for file directories.

verbosity=2
multi=true
monitor.threads=1
monitor.queue=120
monitor.interval=30
fileStore=/var/datahub/filestore
fileStorePurge=/var/datahub/filestorePurge

The monitor.threads controls how many background jobs can run at the same time. Insight tends to get bursts of related information (a number of events and documents in quick succession). If these run at the same time, generated worker and index value rows may be duplicated, depending on the commitFrequency in the Insight datahub.properties. For this reason, it is generally safer to set monitor.threads to 1. In a very busy system, you could experiment with increasing this and decreasing the commitFrequency.

The insight datahub.properties provides properties for Insight itself, including specifying the folder in which the Insight files will be stored, and the prefix to use to retrieve files from the server. The fileURLPrefix is the dhinstance URL.

dbDriver=org.mariadb.jdbc.Driver
dbURL=jdbc:mariadb://127.0.0.1/insight
dbUser=insight
dbPassword=xxxxxxxx
verbosity=3
timestampPrecision=millisecond
commitFrequency=50
load.cacheSize=10000
query.cacheSize=10000
authenticator=com.metrici.datahub.JWTAuthenticator
authorizer=com.metrici.datahub.BasicAuthorizer
fileStore=/var/datahub/filestore/insight
fileURLPrefix=https://service.com/datahub/insight/

Within the Insight configuration folder, copy the public key used within the Insight Hub module (at module/advanced_settings/public_key) into a file called public_key.pem. This means that the data hub will be able to validate the JWTs sent from the Insight Hub module. The public key file should look something like this (but with your key, of course).

Also within the Insight configuration folder, create an authority.json file to describe the permissions required for different users.

{
  "groups": {
    "owner": [
      "ownerUser",
      "null"
    ]
  },
  "permissions": [
    {
      "action": "*",
      "user": "owner",
      "system": "default",
      "entity": "*"
    },
    {
      "action": "*",
      "user": "owner",
      "system": "load",
      "entity": "*"
    },
    {
      "action": "put",
      "user": "*",
      "system": "load",
      "entity": [
        "document_with_worker",
        "event_with_worker",
        "insight_file"
      ]
    },
    {
      "action": "execute",
      "user": "*",
      "system": "search",
      "entity": [
        "dimension",
        "events",
        "documents",
        "worker",
        "document"
      ]
    }
  ]
}

The ownerUser is the install user for the Insight Hub module. The rest should be left as is. The "null" user is required for internal processes that run after the user has been authenticated and authorized, which then run without a user.

default.json, load.json and search.json are the source system definition files included with the Insight Hub release.

Make sure Tomcat has write access to /filestore/insight and /filestorePurge, and that Tomcat is configured to enable it to access the file system.