Ontopic Docs

# Setting up materialization with S3 storage

Starting from Ontopic Studio 2024.1.0

Materialization to RDF is disabled by default (except for the Desktop version, where files are stored locally).

# S3 parameters to collect

# S3_ACCESS_KEY_ID

  • Obtain your S3 access key ID from your AWS account.
  • This key uniquely identifies your account and grants access to your S3 resources.

# S3_ACCESS_KEY_SECRET

  • Retrieve your S3 access key secret (also known as the secret key) from your AWS account.
  • Keep this secret key confidential and secure.

# S3_BUCKET

  • Choose a unique name for your S3 bucket.
  • Buckets are containers for storing objects (files) in S3.

# S3_REGION

  • Determine the AWS region where your S3 bucket will reside.
  • Common regions include us-east-1 (North Virginia), us-west-2 (Oregon), and others.

# Example

  • S3_ACCESS_KEY_ID: AKIAY1234567890
  • S3_ACCESS_KEY_SECRET: mySecretAccessKey
  • S3_BUCKET: my-materialization-bucket
  • S3_REGION: us-west-2

For more detailed information, refer to the Amazon S3 documentation (opens new window).

# Docker-compose

You can enable materialization using the dedicated script.

# Run the script

./enable-materialization.sh

Please provide the S3 parameters when requested. The environment variables will be updated as explained in the next paragraphs.

# Secrets and Configuration Paths

  • It sets the secrets_folder to ./default-secrets or to the legacy folder ./secrets if the latter exists.
  • The following variables are initialized:
    • CONFIG_FILE: Path to the configuration file (./.env by default).
    • SECRETS_DIR: Root directory for secrets (default: ./secrets).
    • VOLUMES_DIR: Root directory for volumes (default: ./volumes).

# Materialization Prompt

  • Asks the user whether to enable materialization.
  • If the answer is "yes" (y), it sets ENABLE_MATERIALIZATION to true.
  • Otherwise, it sets it to false and exits the script.

# S3 Configuration

  • Enters the S3 section.
  • Prompts the user for the S3 access key ID and secret.
  • Writes these secrets to the appropriate files.

# S3 Bucket and Region

  • Prompts the user for the S3 bucket name and region.
  • Writes these configurations.

# Materialization Result and Configuration Directories

  • Sets the paths for materialization result and configuration directories in the default volumes directory.
  • Prints these paths.

The script sets the following environment variables:

  • ENABLE_MATERIALIZATION: Whether materialization is enabled.
  • S3_BUCKET: The S3 bucket name.
  • S3_REGION: The S3 region.
  • MATERIALIZATION_RESULT_DIR: The directory for materialization results.
  • MATERIALIZATION_CONFIGURATION_DIR: The directory for materialization configuration.

The script create two secrets files:

  • s3/access-key-id: The S3 access key ID.
  • s3/access-key-secret: The S3 access key secret.

# AWS Marketplace

Add the materialization configuration using the configuration service running at the port 8080.

Access the tab Studio and click on the checkbox Enable materialization with S3 bucket.

Configure the required fields:

  • Bucket: Specify the name of the S3 bucket where the materialized data will be stored.
  • Region: Enter the AWS region where the S3 bucket is located.
  • Access Key ID: Provide the access key ID associated with your AWS account.
  • Access Key Secret: Enter the corresponding secret for the access key.

Once you’ve filled in the required fields, save the changes. Changes on the configuration service will restart Ontopic Studio.

← Local materialization in Desktop →