Ontopic Docs

# Materialization to S3 storage

Materialization to RDF is disabled by default (except for the Desktop version, where files are stored locally).

# S3 parameters to collect

# S3_ACCESS_KEY_ID

  • Obtain your S3 access key ID from your AWS account.
  • This key uniquely identifies your account and grants access to your S3 resources.

# S3_ACCESS_KEY_SECRET

  • Retrieve your S3 access key secret (also known as the secret key) from your AWS account.
  • Keep this secret key confidential and secure.

# S3_BUCKET

  • Choose a unique name for your S3 bucket.
  • Buckets are containers for storing objects (files) in S3.

# S3_REGION

  • Determine the AWS region where your S3 bucket will reside.
  • Common regions include us-east-1 (North Virginia), us-west-2 (Oregon), and others.

# Example

  • S3_ACCESS_KEY_ID: AKIAY1234567890
  • S3_ACCESS_KEY_SECRET: mySecretAccessKey
  • S3_BUCKET: my-materialization-bucket
  • S3_REGION: us-west-2

For more detailed information, refer to the Amazon S3 documentation (opens new window).

# Docker-compose

You can enable materialization using the dedicated script.

WARNING

Only one storage backend (S3 or Azure Blob Storage) should be configured at a time. Configuring both will result in an error.

# Run the script

./enable-materialization.sh

When prompted, select S3 (option 1) and provide the S3 parameters when requested. The environment variables will be updated as explained in the next paragraphs.

# Secrets and Configuration Paths

  • It sets the secrets_folder to ./default-secrets or to the legacy folder ./secrets if the latter exists.
  • The following variables are initialized:
    • CONFIG_FILE: Path to the configuration file (./.env by default).
    • SECRETS_DIR: Root directory for secrets (default: ./secrets).
    • VOLUMES_DIR: Root directory for volumes (default: ./volumes).

# Materialization Prompt

  • Asks the user whether to enable materialization.
  • If the answer is "yes" (y), it sets ENABLE_MATERIALIZATION to true.
  • Otherwise, it sets it to false and exits the script.

# Storage Backend Selection

  • Prompts the user to select between S3 (option 1) or Azure Blob Storage (option 2).
  • Select option 1 for S3.

# S3 Configuration

  • Enters the S3 section.
  • Prompts the user for the S3 access key ID and secret.
  • Writes these secrets to the appropriate files.

# S3 Bucket and Region

  • Prompts the user for the S3 bucket name and region.
  • Writes these configurations.

# Materialization Result and Configuration Directories

  • Sets the paths for materialization result and configuration directories in the default volumes directory.
  • Prints these paths.

The script sets the following environment variables:

  • ENABLE_MATERIALIZATION: Whether materialization is enabled.
  • S3_BUCKET: The S3 bucket name.
  • S3_REGION: The S3 region.
  • MATERIALIZATION_RESULT_DIR: The directory for materialization results.
  • MATERIALIZATION_CONFIGURATION_DIR: The directory for materialization configuration.

The script create two secrets files:

  • s3/access-key-id: The S3 access key ID.
  • s3/access-key-secret: The S3 access key secret.

# Kubernetes

# Store S3 credentials as files

First, save your S3 credentials to local files:

# Create secrets directory if it doesn't exist
mkdir -p ./secrets
echo -n "<S3_ACCESS_KEY_ID>" > ./secrets/s3-access-key-id
echo -n "<S3_ACCESS_KEY_SECRET>" > ./secrets/s3-access-key-secret

# Create Kubernetes secrets

Use kubectl to create secrets from the credential files:

kubectl create secret generic s3-access-key-id \
  --from-file=s3-access-key-id=./secrets/s3-access-key-id
kubectl create secret generic s3-access-key-secret \
  --from-file=s3-access-key-secret=./secrets/s3-access-key-secret

# Configure Helm values

Update your Helm values.yaml file with the S3 bucket configuration:

ontopic-server:
  enableMaterialization: true
  objectStorage:
    s3:
      bucket: <S3_BUCKET>
      region: <S3_REGION>
      endpoint: # Optional, default is https://s3.amazonaws.com

# Apply changes

To apply changes, stop and start Ontopic Suite again.

# AWS Marketplace

Add the materialization configuration using the configuration service running at the port 8080.

Access the tab Studio and click on the checkbox Enable materialization with S3 bucket.

Configure the required fields:

  • Bucket: Specify the name of the S3 bucket where the materialized data will be stored.
  • Region: Enter the AWS region where the S3 bucket is located.
  • Access Key ID: Provide the access key ID associated with your AWS account.
  • Access Key Secret: Enter the corresponding secret for the access key.

Once you’ve filled in the required fields, save the changes. Changes on the configuration service will restart Ontopic Suite.

← Materialization to Azure Blob Storage →