Skip to content

You are viewing documentation for Immuta version 2023.3.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

cancel

Data Source Payload Attribute Details

Audience: Data Engineers

Content Summary: This page contains details and examples of payload attributes for creating data sources.

connectionKey

The connectionKey is a unique identifier for the collection of data sources being created. If an existing connectionKey is used with new connection information, it will delete the old data sources and create new ones from the new information in the payload.

Data source duplicates

In order to avoid two data sources referencing the same table, users can not create duplicate data sources. If you attempt to create a duplicate data source using different connectionKeys and the same hostname and database, you will encounter a warning stating "You may only define data sources for the same connection once."

connection

Attribute Description
handler Databricks, Google BigQuery, Presto, Redshift, Snowflake, and Trino.
ssl boolean Set to true to enable SSL communication with the remote database.
database string The database name.
schema string The schema in the remote database.
userFiles array Array of objects; each object must have keyName (corresponds to an ODBC connection string option), content (base-64 encoded content), and userFilename (the name of the file - for display purposes in the app).
connectionStringOptions string Additional ODBC connection string options to be used when connecting to the remote database.
hostname string The hostname of the remote database instance.
port number The port of the remote database instance.
authenticationMethod See special cases below.
username string The username used to connect to the remote database.
password string The password used to connect to the remote database.

Special Cases

  • Athena: Also requires region and queryResultLocationBucket. queryResultLocationDirectory is optional. authenticationMethod can be none, accessKey (default: username = access key, password = secret key), or instanceRole.
  • BigQuery: Does not require hostname and password. Requires sid, which is the GCP project ID, and userFiles with the keyName of KeyFilePath and the base64-encoded keyfile.json.
  • Databricks: Also requires httpPath. No username is required.
  • Hadoop: authenticationMethod can be none, userPassword, hdInsight, kerberos, kerberosHdInsight or emr.
  • Trino: authenticationMethod can be No Authentication, LDAP Authentication, or Kerberos Authentication.
  • Snowflake: Also requires warehouse. authenticationMethod can be userPassword or PRIV_KEY_FILE. If using PRIV_KEY_FILE, do not specify a password; userFiles is required with the keyName of PRIV_KEY_FILE and the base64-encoded Snowflake key.

nameTemplate

Attribute Description
dataSourceFormat string Format to be used to name the data sources created in this group.
schemaFormat string Format to be used to name the Immuta schema created in this group.
tableFormat string Format to be used to name the Immuta table created in this group.
schemaProjectNameFormat string Format to be used to name the Immuta schema project created in this group.

Available templates include

  • <tablename>
  • <schema>
  • <database>

All cases of the name in Immuta should be lowercase.

For example, consider a table TPC.CUSTOMER that is given the following nameTemplate:

dataSourceFormat: <schema> <tablename>
tableFormat: <tablename>
schemaFormat: <schema>
schemaProjectNameFormat: <schema>

This nameTemplate will produce a data source named tpc.customer in a schema project named tpc.

options

Attribute Description
staleDataTolerance integer The length in seconds that data for these sources can be cached.
expiration date Date that the data source should be purged from Immuta. Defaults to no expiration.
disableSensitiveDataDiscovery boolean If true, Immuta will not perform Sensitive Data Discovery. Default: false.
hardDelete boolean If true, when the table backing the data source is no longer available, the data source in Immuta is deleted. If this is false, the data source will be disabled. Default: false.
tableTags array An array of tags (strings) to place at the data source level on every data source.

owners

Attribute Description
type group or user The type of owner that is being added.
name string The name of the group or the user (username they log in with).
iam (optional) string The ID of the identity manager system the user or group comes from. If excluded, any user/group that matches will be added as an owner.

sources

Best Practice: Use Subscription Policies to Control Access

If you are not tagging individual columns, omit sources to create data sources for all tables in the schema or database, and then use Subscription Policies to control access to the tables instead of excluding them from Immuta.

This attribute configures which sources are created. If sources is not provided, all sources from the given connection will be created.

There are 3 types of sources than can be specified:

Specify a Table

If you want to select specific tables to be created as data sources, or if you want to tag individual data sources or columns within a data source, you need to leverage this section:

sources:
  - table: name_of_table
    schema: name_of_schema

Specify a Query

Immuta recommends creating a view in your native database instead of using this option, but if that is not possible, you can create data sources based on SQL statements:

sources:
  - query: “select * from table”
    naming:
      datasource: “My Source”,
      table: “my_source”,
      schema: “queries”

Specify All Tables

If you specify any sources (either tables or queries), but you still want to create data sources for the rest of the tables in the schema/database, you can specify all as a source:

sources:
  - all: true

Schema Monitoring

Excluding sources or specifying all: true will turn on automatic schema monitoring in Immuta. As tables are added or removed, Immuta will look for those changes on a schedule (by default, once a day) and either disable/delete data sources for removed tables or create data sources for new tables. New tables will be tagged New so that you can build a policy to restrict access to new tables until they are evaluated by data owners. Data owners will be notified of new tables, and all subscribers will be notified if data sources are disabled or deleted.

Additional Options

When specifying a table or query there are other options that can be specified:

Option Description
columnDescriptions See details below.
description A short description for the data source.
documentation Markdown-supported documentation for the data source.
naming See the example above in Specify a Query. This is required for query-based sources, but is optional for table-based sources and can be used to override the nameTemplate provided for the whole database/schema.
owners Specify owners for an individual data source. The payload is the same as owners at the root level.
tags See details below.

Columns

Columns

  • If any columns are specified, those are the only columns that will be available in the data source.
  • If no columns are specified, Immuta will look for new or removed columns on a schedule (by default, once a day) and add or remove columns from the data sources automatically as needed.
  • New columns will be tagged New, so you can build a policy to automatically mask new columns until they are approved.
  • Data Owners will be notified when columns are added or removed.

columns is an array of objects for each column:

Attribute Description
name The column name.
dataType The Postgres data type for the Immuta Query Engine.
nullable Whether or not the column contains null.
remoteType The actual data type in the remote database.
primaryKey Specify whether this is the primary key of the remote table.
description Describe the column.

Column Descriptions

You can add descriptions to columns without having to specify all the columns in the data source. columnDescriptions is an array of objects with the following schema:

Attribute Description
columnName string The column name.
description string The description of the column. 
columnDescriptions:
  - columnName: acct_num
    description: The account number

Tags

You can add tags to columns or data sources. tags is an object with the following schema:

Attribute Description
table array An array of tags (strings) to add to this table.
columns array An array of objects that specifies columnName (string) and tags (an array to tags). 
tags:
  table:
    - Sensitive
    - Marketing
  columns:
    - columnName: acct_num
      tags:
        - unique_id