Configuring ClickHouse source endpoints

• Endpoint type: clickhouse-source.

• --cluster-id: ID of the cluster you need to connect to.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

  Make sure the specified security groups are configured.

• To set a user password to access the DB, use one of the following parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: clickhouse_source.

• connection.connection_options.mdb_cluster_id: ID of cluster to connect to.

• subnet_id: ID of the subnet hosting the cluster. If not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• security_groups: Specify the security groups for network traffic.

  This will let you apply the specified security group rules to the VMs and clusters in the subnet_id network without changing the settings of these VMs and clusters. For more information, see Network in Data Transfer.

  Make sure the specified security groups are configured.

• connection.connection_options.database: Database name.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint name in Terraform>" {
  name = "<endpoint name>"
  settings {
    clickhouse_source {
      security_groups = [ "list of security group IDs" ]
      subnet_id       = "<subnet ID>"
      connection {
        connection_options {
          mdb_cluster_id = "<Managed Service for ClickHouse cluster ID>"
          database       = "<name of database to transfer>"
          user           = "<username to connect>"
          password {
            raw = "<user password>"
          }
        }
      }
      <advanced endpoint settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

  Make sure the specified security groups are configured.

• mdbClusterId: ID of the cluster you need to connect to.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• Endpoint type: clickhouse-source.

• --host: IP address or FQDN of the master host you want to connect to.

• --port: Number of the port that Data Transfer will use for the connection.

• --ca-certificate — If the transmitted data needs to be encrypted, for example, to meet the requirements of PCI DSS.

• --subnet-id: ID of the subnet the host resides in.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: clickhouse_source.

• Shard settings:

  • connection.connection_options.on_premise.shards.name: Shard name that the service will use to distinguish shards from each other.
  • connection.connection_options.on_premise.shards.hosts: specify the FQDNs or IP addresses of the hosts in the shard.

• connection.connection_options.on_premise.http_port: Port number that Data Transfer will use for HTTP connections.

• connection.connection_options.on_premise.native_port: Port number that Data Transfer will use for connections to the ClickHouse native interface.

• connection.connection_options.on_premise.tls_mode.enabled.ca_certificate: CA certificate used if the data being transferred must be encrypted to comply with the PCI DSS requirements.

• subnet_id: ID of the subnet hosting the cluster. If not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• security_groups: Specify the security groups for network traffic.

  This will let you apply the specified security group rules to the VMs and clusters in the subnet_id network without changing the settings of these VMs and clusters. For more information, see Network in Data Transfer.

• connection.connection_options.database: Database name.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint name in Terraform>" {
  name = "<endpoint name>"
  settings {
    clickhouse_source {
      security_groups = [ "list of security group IDs" ]
      subnet_id       = "<subnet ID>"
      connection {
        connection_options {
          on_premise {
            http_port   = "<HTTP connection port>"
            native_port = "<native interface connection port>"
            shards {
              name  = "<shard name>"
              hosts = [ "shard host IP or FQDN list" ]
            }
            tls_mode {
              enabled {
                ca_certificate = "<certificate in PEM format>"
              }
            }
          }
          database = "<name of database being transferred>"
          user     = "<username for connection>"
          password {
            raw = "<user password>"
          }
        }
      }
      <advanced endpoint settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• onPremise: Database connection parameters:

  • hosts — IP address or FQDN of the master host to connect to.

  • port: The number of the port that Data Transfer will use for the connection.

  • tlsMode: Parameters of encryption of transmitted data if it is required, for example, to meet the requirements of PCI DSS.

  • subnetId: ID of the subnet the host resides in.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• include_tables: List of included tables. If this is on, the data will only be transferred from the tables in this list.

  When you add new tables when editing an endpoint used in Snapshot and increment or Replication transfers with the Replicating status, the data history for these tables will not get uploaded. To add a table with its historical data, use the List of objects to be transferred (Preview) field in the transfer settings.

• exclude_tables: List of excluded tables. Data from tables on this list will not be transmitted.

The lists include the name of the schema (description of DB contents, structure, and integrity constraints) and the table name. Both lists support expressions in the following format:

• <schema name>.<table name>: Fully qualified table name.
• <schema name>.*: All tables in the specified schema.
• <table name>: Table in the default schema.