Storage Target Configuration

DataCore Software Ngenea HSM supports Storage Targets compatible with the Amazon S3 protocol, Spectra Logic's BlackPearl Spectra S3 protocol (a superset of the Amazon S3 Protocol), Microsoft Azure and Google Cloud.

Storage Targets are referenced from the Master Configuration File, with configuration data for each target defined in a target-specific configuration file.

The default location for a Storage Target Configuration File is in /opt/arcapix/etc/.

The following subsections describe formats of configuration files for supported storage target types:

Multi-Target Support

The following section describes how to configure Ngenea HSM to migrate and recall to multiple Storage Targets in parallel.

With reference to the Master Configuration File example in General Configuration which demonstrates a configuration to migrate data from two differing directories to associated targets respectively, Ngenea HSM can migrate data from the same directory or even specific individual files to multiple targets.

Such a configuration can help ensure data can be recalled if a storage target is unavailable, for instance due to a disaster or a provider service outage.

Example Whole Filesystem to Multiple-Targets Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
[General]
MinMigrateEndpointCount=0

[Storage awss3]
StorageType=AmazonS3
StorageKey=01
ConfigFile=awss3.conf
LocalFileRegex=/mmfs1/(.+)
RemoteLocationXAttrRegex=awss3:(.+)

[Storage azure]
StorageType=Azure
StorageKey=02
ConfigFile=azure.conf
RemoteLocationXAttrRegex=azure:(.+)
LocalFileRegex=/mmfs1/(.+)

File Matching to Multiple Targets

Ngenea HSM matches a file to be migrated against the LocalFileRegex parameter of each Storage Target to determine if the file should be migrated to the target. To migrate files to multiple storage target endpoints ensure that the LocalFileRegex either specifically targets or overlaps the file's path for each relevant storage target.

As per the above example, specifying LocalFileRegex=/mmfs1/(.+) for all targets will ensure that all files within the /mmfs1 directory tree are migrated to both targets.

Alternatively, more granular control can be applied.

Given two storage targets (A) and (B) with differing values for LocalFileRegex parameters:

A: LocalFileRegex=/mmfs1/(.+)

B: LocalFileRegex=/mmfs1/archive/(.+)

After successful migration the first Storage Target (A) would contain all files under the /mmfs1 directory structure. The second Storage Target (B) will only contain files within the /mmfs1/archive directory structure. Thus resulting in identically migrated files within the /mmfs1/archive directory structure on both Storage Targets.

Recall Behaviour

Ngenea HSM will recall migrated files in top-down order of the Storage Target stanzas in the Master Configuration File which match the criteria for the file. Should a storage target not be able to provide the file for recall, Ngenea HSM will use the next matching Storage Target in top-down order - and so on, until all Storage Targets have been exhausted.

Multi-Target Specific Configuration Keywords

  • MinMigrateEndpointCount

  • StorageKey

MinMigrateEndpointCount

Defines the number of associated endpoints a file needs to be migrated to in order for the migration to be considered a success.

If the "MinMigrateEndpointCount" parameter is equal to 0 or is greater than or equal to the number of associated storage endpoints for the file, and the file was successfully migrated to all those endpoints, then Ngenea HSM considers the file migrated successfully.

If the "MinMigrateEndpointCount" parameter is greater than 0 and is less than the number of associated storage endpoints for file, and the file was successfully migrated to the minimum number of endpoints as defined by "MinMigrateEndpointCount", then Ngenea HSM considers the file migrated successfully.

If the file was successfully migrated to at least one endpoint associated with the file but less than the minimum number of storage endpoints as defined by "MinMigrateEndpointCount", then Ngenea HSM considers the file migration partially successful.

If the file was not successfully migrated to any endpoint, Ngenea HSM considers the migration of the file as failed.

On exit, Ngenea HSM ngmigrate utility returns the following exit statuses:

  • 0 - all files were migrated successfully

  • 1 - all files failed to migrate successfully

  • 2 - some files failed to migrate successfully

Syntax

MinMigrateEndpointCount=<Non-negative integer value>

Example

MinMigrateEndpointCount=2

StorageKey

Defines a custom identifier attributable to the storage endpoint.

Every [Storage] section of the Master Configuration File can contain the "StorageKey" parameter. The parameter's value must be a two-byte string not equal to the characters "sz". The default value of the parameter is "lc".

When implementing multi-target migration for files, each storage endpoint must have the parameter defined with a unique key.

Syntax

StorageKey=<char><char>

Examples

StorageKey=16

StorageKey=zz

StorageKey=3a