Configuring a ClickHouse target endpoint

• Endpoint type: clickhouse-target.

• --cluster-id: ID of the cluster you need to connect to.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

  Make sure the specified security groups are configured.

• To set a user password to access the DB, use one of the following parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: clickhouse_target.

• connection.connection_options.mdb_cluster_id: ID of cluster to connect to.

• subnet_id: ID of the subnet hosting the cluster. If not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• security_groups: Specify the security groups for network traffic.

  This will let you apply the specified security group rules to the VMs and clusters in the subnet_id network without changing the settings of these VMs and clusters. For more information, see Network in Data Transfer.

  Make sure the specified security groups are configured.

• connection.connection_options.database: Database name.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint name in Terraform>" {
  name = "<endpoint name>"
  settings {
    clickhouse_target {
      security_groups = [ "list of security group IDs" ]
      subnet_id       = "<subnet ID>"
      connection {
        connection_options {
          mdb_cluster_id = "<Managed Service for ClickHouse cluster ID>"
          database       = "<name of database to transfer>"
          user           = "<username to connect>"
          password {
            raw = "<user password>"
          }
        }
      }
      <advanced endpoint settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

  Make sure the specified security groups are configured.

• mdbClusterId: ID of the cluster you need to connect to.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• Endpoint type: clickhouse-target.

• --host: IP address or FQDN of the master host you want to connect to.

• --port: Number of the port that Data Transfer will use for the connection.

• --ca-certificate — If the transmitted data needs to be encrypted, for example, to meet the requirements of PCI DSS.

• --subnet-id: ID of the subnet the host resides in.

• --database: Database name.

• --user: Username that Data Transfer will use to connect to the database.

• --security-group: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

• To set a user password to access the database, use one of the parameters:

  • --raw-password: Password as text.

  • --password-file: The path to the password file.

• Endpoint type: clickhouse_target.

• Shard settings:

  • connection.connection_options.on_premise.shards.name: Shard name that the service will use to distinguish shards from each other.
  • connection.connection_options.on_premise.shards.hosts: specify the FQDNs or IP addresses of the hosts in the shard.

• connection.connection_options.on_premise.http_port: Port number that Data Transfer will use for HTTP connections.

• connection.connection_options.on_premise.native_port: Port number that Data Transfer will use for connections to the ClickHouse native interface.

• connection.connection_options.on_premise.tls_mode.enabled.ca_certificate: CA certificate used if the data being transferred must be encrypted to comply with the PCI DSS requirements.

• subnet_id: ID of the subnet hosting the cluster. If not specified, the cluster must be accessible from the internet.

  If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

• security_groups: Specify the security groups for network traffic.

  This will let you apply the specified security group rules to the VMs and clusters in the subnet_id network without changing the settings of these VMs and clusters. For more information, see Network in Data Transfer.

• connection.connection_options.database: Database name.

• connection.connection_options.user: Username that Data Transfer will use to connect to the database.

• connection.connection_options.password.raw: Password in text form.

Example of the configuration file structure:

resource "yandex_datatransfer_endpoint" "<endpoint name in Terraform>" {
  name = "<endpoint name>"
  settings {
    clickhouse_target {
      security_groups = [ "list of security group IDs" ]
      subnet_id       = "<subnet ID>"
      connection {
        connection_options {
          on_premise {
            http_port   = "<HTTP connection port>"
            native_port = "<native interface connection port>"
            shards {
              name  = "<shard name>"
              hosts = [ "list of shard host IPs or FQDNs" ]
            }
            tls_mode {
              enabled {
                ca_certificate = "<certificate in PEM format>"
              }
            }
          }
          database = "<name of database to transfer>"
          user     = "<username to connect>"
          password {
            raw = "<user password>"
          }
        }
      }
      <advanced endpoint settings>
    }
  }
}

For more information, see the Terraform provider documentation.

• onPremise: Database connection parameters:

  • hosts — IP address or FQDN of the master host to connect to.

  • port: The number of the port that Data Transfer will use for the connection.

  • tlsMode: Parameters of encryption of transmitted data if it is required, for example, to meet the requirements of PCI DSS.

  • subnetId: ID of the subnet the host resides in.

• securityGroups: Security groups for network traffic, whose rules will apply to VMs and clusters without changing their settings. For more information, see Network in Data Transfer.

• database: Database name.

• user: Username that Data Transfer will use to connect to the database.

• password.raw: Database user password (in text form).

• clickhouse_cluster_name: Specify the name of the cluster that the data will be transferred to.

• alt_names: If necessary, set rules for renaming the source database tables when transferring them to the target database:

  • from_name: Source table name.
  • to_name: Target table name.

• sharding.column_value_hash.column_name: The name of the column in tables that data will be sharded by. A uniform distribution across shards will be determined by a hash from this column value.

• sharding.transfer_id: If true, the data is sharded based on the transfer ID value. The transfer will ignore the sharding.column_value_hash.column_name setting and will only shard the data based on the transfer ID.

  Warning

  If you omit the sharding columns and the sharding.transfer_id setting, all the data will be moved to the same shard.

• cleanup_policy: Select a way to clean up data in the target database before the transfer:

  • CLICKHOUSE_CLEANUP_POLICY_DROP: Fully delete tables included in the transfer (default).

    Use this option so that the latest version of the table schema is always transferred to the target database from the source whenever the transfer is activated.

  • CLICKHOUSE_CLEANUP_POLICY_DISABLED: Do not clean up.

    Select this option if only replication without copying data is performed.