How to configure Conduit

Conduit accepts CLI flags, environment variables and a configuration file to configure its behavior. Each CLI flag has a corresponding environment variable and a corresponding field in the configuration file. Conduit uses the value for each configuration option based on the following priorities:

CLI flags

CLI flags (highest priority) - if a CLI flag is provided it will always be respected, regardless of the environment variable or configuration file. To see a full list of available flags run conduit run --help:

Starts the Conduit server and runs the configured pipelines.

Usage:
  conduit run [flags]

Flags:
      --api.enabled                                            enable HTTP and gRPC API (default true)
      --api.grpc.address string                                address for serving the gRPC API (default ":8084")
      --api.http.address string                                address for serving the HTTP API (default ":8080")
      --config.path string                                     global conduit configuration file (default "/Users/username/repo/conduit/conduit.yaml")
      --connectors.max-receive-record-size int                 maximum size of a processed record in bytes for standalone connectors. (default 4194304)
      --connectors.path string                                 path to standalone connectors' directory (default "/Users/username/repo/conduit/connectors")
      --db.badger.path string                                  path to badger DB (default "/Users/username/repo/conduit/conduit.db")
      --db.postgres.connection-string string                   postgres connection string, may be a database URL or in PostgreSQL keyword/value format
      --db.postgres.table string                               postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.sqlite.path string                                  path to sqlite3 DB (default "/Users/username/repo/conduit/conduit.db")
      --db.sqlite.table string                                 sqlite3 table in which to store data (will be created if it does not exist) (default "conduit_kv_store")
      --db.type string                                         database type; accepts badger,postgres,inmemory,sqlite (default "badger")
      --dev.blockprofile string                                write block profile to file
      --dev.cpuprofile string                                  write CPU profile to file
      --dev.memprofile string                                  write memory profile to file
  -h, --help                                                   help for run
      --log.format string                                      sets the format of the logging; accepts json, cli (default "cli")
      --log.level string                                       sets logging level; accepts debug, info, warn, error, trace (default "info")
      --pipelines.error-recovery.backoff-factor int            backoff factor applied to the last delay (default 2)
      --pipelines.error-recovery.max-delay duration            maximum delay before restart (default 10m0s)
      --pipelines.error-recovery.max-retries int               maximum number of retries (default -1)
      --pipelines.error-recovery.max-retries-window duration   amount of time running without any errors after which a pipeline is considered healthy (default 5m0s)
      --pipelines.error-recovery.min-delay duration            minimum delay before restart (default 1s)
      --pipelines.exit-on-degraded                             exit Conduit if a pipeline is degraded
      --pipelines.path string                                  path to pipelines' directory (default "/Users/username/repo/conduit/pipelines")
      --preview.pipeline-arch-v2                               enables experimental pipeline architecture v2 (note that the new architecture currently supports only 1 source and 1 destination per pipeline)
      --preview.pipeline-arch-v2-disable-metrics               disables metrics about amount of data (in bytes) moved in pipeline architecture v2 (increases performance)
      --processors.path string                                 path to standalone processors' directory (default "/Users/username/repo/conduit/processors")
      --schema-registry.confluent.connection-string string     confluent schema registry connection string
      --schema-registry.type string                            schema registry type; accepts builtin,confluent (default "builtin")

To know the current configurations that conduit will use when you run it, run conduit config:

$ ./conduit config
config.path: /Users/username/repo/conduit/conduit.yaml
db.type: badger
db.badger.path: /Users/username/repo/conduit/conduit.db
db.postgres.table: conduit_kv_store
db.sqlite.path: /Users/username/repo/conduit/conduit.db
db.sqlite.table: conduit_kv_store
api.enabled: true
api.http.address: :8080
api.grpc.address: :8084
log.level: info
log.format: cli
connectors.path: /Users/username/repo/conduit/connectors
connectors.max-receive-record-size: 4194304
processors.path: /Users/username/repo/conduit/processors
pipelines.path: /Users/username/repo/conduit/pipelines
pipelines.exit-on-degraded: false
pipelines.error-recovery.min-delay: 1s
pipelines.error-recovery.max-delay: 10m0s
pipelines.error-recovery.backoff-factor: 2
pipelines.error-recovery.max-retries: -1
pipelines.error-recovery.max-retries-window: 5m0s
schema-registry.type: builtin
preview.pipeline-arch-v2: false
preview.pipeline-arch-v2-disable-metrics: false

Environment variables

Environment variables (lower priority) - an environment variable is only used if no CLI flag is provided for the same option. Every setting is configurable this way: there is no flag without a matching environment variable. The name is derived from the flag path mechanically:

1. Take the flag path (e.g. api.http.address).
2. Upper-case it and replace every . and - with _.
3. Prefix it with CONDUIT_.

Flag	Environment variable
--api.http.address	CONDUIT_API_HTTP_ADDRESS
--api.enabled	CONDUIT_API_ENABLED
--db.type	CONDUIT_DB_TYPE
--db.postgres.connection-string	CONDUIT_DB_POSTGRES_CONNECTION_STRING
--log.level	CONDUIT_LOG_LEVEL

For example, these two invocations are equivalent:

$ conduit run --api.http.address=:9090

$ CONDUIT_API_HTTP_ADDRESS=:9090 conduit run

note

Because a flag always wins over an environment variable, a flag on the command line overrides the same setting exported in your environment. See precedence above.

Configuration file

Configuration file (lowest priority) - Conduit by default loads the file conduit.yaml placed in the same folder as Conduit. The path to the file can be customized using the CLI flag --config.path. It is not required to provide a configuration file and any value in the configuration file can be overridden by an environment variable or a flag. The file content should be a YAML document where keys can be hierarchically split on .. For example:

db:
  type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE
  postgres:
    connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING

Pointing Conduit at a pipelines directory

The path Conduit scans for pipeline configuration files is set by --pipelines.path (env CONDUIT_PIPELINES_PATH). For convenience — and to make GitOps-style setups read cleanly — conduit run also accepts --pipelines as a shorthand alias for --pipelines.path:

$ conduit run --pipelines ./my-pipelines

This is equivalent to conduit run --pipelines.path ./my-pipelines. The argument can be a directory of pipeline configuration files or a single pipeline configuration file.

Secrets in logs

Connector settings routinely carry secrets: database URLs with embedded passwords, SASL credentials, access keys. Conduit redacts connector configuration values in its own logs — when it logs a connector's config (for example while configuring a connector), every value is replaced with *** and only the parameter keys stay visible. Keys are not secret and are useful for debugging ("which parameter is set") without revealing the value it was set to.

note

This redaction applies to Conduit's own log output. It does not alter the values Conduit passes to the connector, and it is independent of any redaction a connector performs on its own logs.