Both documents and events are associated with files. In each case, the files are represented by a FILE_INFO data structure.
In normal operation, Insight Hub writes a record of each file to:
insightFolder.files.instanceFolder.file
Where
- insightFolder is the install folder for Insight
- instanceFolder is a folder for the instance
- file is a local reference built from the file UUID.
A modified FILE_INFO, which includes a URL, is then sent to the data hub. This is stored within the insight_file table. The uuid is used to reference the file in document and event loads.
When data is retrieved from the hub, the FILE_INFO written to the hub is returned. This is used to look up the record of the file, and to compose a full FILE_INFO block that the client can use to read the file.
Uploads
The write of insight_file data to the data hub is compatible with a direct file upload to the data hub. The file_uuid is passed as a property in the options string, and the authenticated user is used as the instance uuid.
A file can be uploaded to the data hub using options of form.
{
"file_uuid": "fileuuid"
}And the uploaded using something like the following curl call, which has been split over multiple lines for clarity.
curl -X POST
-F "options=<options.json"
-F "data=@path_to_file.ext"
-H "File-Name:your_file_name.ext"
-H "Authorization: Bearer xxxxxxxx"
"dhinstance/load/insight_file/file?process=true"
The "<options.json" clause reads a field value from a text file, the" @path_to_file.ext" reads a file and encodes it as a file on the HTTP message.
The bearer token is a JWT. This can be generated from the Insight module server to authenticate the instance user.
Reading files by URL
Files sent from the process module to the data hub have the URL field populated, and should be accessible on the Insight module user. The instance user will need a JWT generated from the server to access these.
Files uploaded directly to the data hub will have a URL. However, the file uuid will not match one in the module. In this case, the uuid in the FILE_INFO will be removed and the URL sent to the client, which can then rertieve it directly from the data hub. The data hub generates long and secure unique URLs and provides unauthenticated access to these, and they should be readable by the client. Note that this usage has not yet been fully explored and should be considered an early experimental feature.