[Bug]: Unable to run on the Clickhouse Cloud #4316
Description
What component(s) are affected?
- Opik Python SDK
- Opik Typescript SDK
- Opik Agent Optimizer SDK
- Opik UI
- Opik Server
- Documentation
Opik version
- Opik version:
apiVersion: v2
name: opik
type: application
version: 1.9.36
appVersion: 1.9.36
Describe the problem
Hey, I'm trying to set up Opik to run with the Clickhouse Cloud we're using in the company. Unfortunately, it's failing during the migration process.
Some of the values provided to the chart:
env:
GENERAL_LOG_LEVEL: DEBUG
OPIK_LOG_LEVEL: DEBUG
STATE_DB_PROTOCOL: "jdbc:mysql://"
STATE_DB_URL: "gcloud-sqlproxy.routing.svc.cluster.local:3306/opik?currentSchema=opik&rewriteBatchedStatements=true"
MYSQL_USE_SSL: "true"
STATE_DB_DATABASE_NAME: "opik"
STATE_DB_USER: "opik-dbuser"
ANALYTICS_DB_MIGRATIONS_URL: "jdbc:clickhouse://xxxx.europe-west4.gcp.clickhouse.cloud:8443/opik?ssl=true&?currentSchema=opik"
ANALYTICS_DB_HOST: "xxxx.europe-west4.gcp.clickhouse.cloud"
ANALYTICS_DB_PROTOCOL: "HTTPS" #changed
ANALYTICS_DB_PORT: "8443" #changed
ANALYTICS_DB_DATABASE_NAME: "opik" #changed
ANALYTICS_DB_MIGRATIONS_DATABASE_NAME: "opik"
ANALYTICS_DB_MIGRATIONS_USER: "opik-rw"
ANALYTICS_DB_USERNAME: "opik-rw"
The error log:
Caused by: liquibase.exception.DatabaseException: ClickHouse exception, code: 1002, host: xxxxxx.europe-west4.gcp.clickhouse.cloud, port: 8443; Code: 36. DB::Exception: It's not allowed to specify explicit zookeeper_path and replica_name for ReplicatedMergeTree arguments in Rep │
│ [Failed SQL: (1002) CREATE TABLE IF NOT EXISTS opik.automation_rule_evaluator_logs1 │
│ ( │
│ `timestamp` DateTime64(9, 'UTC') DEFAULT now64(9), │
│ `workspace_id` String, │
│ `rule_id` FixedString(36), │
│ `level` Enum8('TRACE' = 0, 'DEBUG' = 1, 'INFO' = 2, 'WARM' = 3, 'ERROR' = 4), │
│ `message` String, │
│ `markers` Map(String, String) │
│ ) │
│ ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/opik/automation_rule_evaluator_logs', '{replica}') │
│ ORDER BY (workspace_id, rule_id, timestamp) │
│ TTL toDateTime(timestamp + toIntervalMonth(6)) │
│ SETTINGS index_granularity = 8192] │
│ at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:520) │
│ at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:85) │
│ at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:188) │
│ at liquibase.executor.AbstractExecutor.execute(AbstractExecutor.java:148) │
│ at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1198) │
│ at liquibase.changelog.ChangeSet.execute(ChangeSet.java:816) │
│ ... 43 more │
│ Caused by: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: xxxxx.europe-west4.gcp.clickhouse.cloud, port: 8443; Code: 36. DB::Exception: It's not allowed to specify explicit zookeeper_path and replica_name for ReplicatedMergeTree │
│ │
│ at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.getException(ClickHouseExceptionSpecifier.java:92) │
│ at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:56) │
│ at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:29) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(ClickHouseStatementImpl.java:1094) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:773) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.getLastInputStream(ClickHouseStatementImpl.java:691) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:340) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:324) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:319) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:314) │
│ at ru.yandex.clickhouse.ClickHouseStatementImpl.execute(ClickHouseStatementImpl.java:445) │
│ at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) │
│ at java.base/java.lang.reflect.Method.invoke(Method.java:580) │
│ at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:123) │
│ at jdk.proxy2/jdk.proxy2.$Proxy90.execute(Unknown Source) │
│ at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:512) │
│ ... 48 more │
│ Caused by: java.lang.Throwable: Code: 36. DB::Exception: It's not allowed to specify explicit zookeeper_path and replica_name for ReplicatedMergeTree arguments in Replicated database. If you really want to specify them explicitly, enable setting database_replicated_allow_replicated │
│ │
│ at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:54) │
│ ... 62 more
From what I've checked, replicated table engines in the Clickhouse cloud are managed by the provider:
https://clickhouse.com/docs/engines/table-engines/mergetree-family/replication
And for that reason, with the current SQL migrations, it's impossible to start the opik on the Clickhouse cloud - which is a shame :<
I can try to do some hacks with the migrations, but it would be pretty nice to have an option to install opik there
Reproduction steps and code snippets
With the latest helm chart, try to install opik and point to the Clickhouse database running in the Clickhouse cloud
Error logs or stack trace
No response
Healthcheck results
No response