Skip to main content

Conduit v0.21.0 release

· 3 min read

Conduit v0.21.0 is the first release aimed squarely at teams moving off Kafka Connect. It adds conduit migrate kafka-connect, which turns a Kafka Connect worker and connector config into a Conduit pipeline plus a compatibility report, and it certifies a first set of connectors for that path: Postgres CDC, MySQL CDC (beta), Elasticsearch/OpenSearch, Kafka→Kafka mirroring, and the S3 sink. It also lands lossless structured values over protocol v3, Protobuf decode, secrets-at-rest, the processor batch API, and the Python connector SDK GA.

Migrating from Kafka Connect​

  • conduit migrate kafka-connect. Reads a Kafka Connect worker and connector config and emits Conduit pipeline config plus a compatibility report. Every setting is translated, approximated, or flagged as blocking — the command never silently drops config it can't translate. It covers Debezium Postgres and MySQL, JDBC source and sink, the S3 sink, Elasticsearch and OpenSearch, and Kafka→Kafka (MirrorMaker-style).
  • No re-snapshot for Postgres. A migrated Postgres pipeline resumes from its Debezium offsets: Conduit creates its own replication slot (the fresh-slot strategy) at the imported position. v0.21 ships that path for Postgres only; migrations of the other connectors still re-snapshot, and offset import for them is v0.22.
  • Debezium-compatible output for Postgres. An opt-in output mode emits the envelope, topic naming, key schema, tombstones, and decimal/time encodings that Debezium consumers expect, verified against real Debezium on the same database.

Certified connectors​

Every connector conduit migrate kafka-connect emits is certified in this release.

  • Postgres CDC — Debezium parity plus offset import (fresh-slot), chaos-tested.
  • MySQL CDC (beta) — the labs mysql source, certified as beta.
  • Elasticsearch and OpenSearch — the sink, certified from labs.
  • Kafka→Kafka — mirroring with partition, header, and topic preservation, plus a Redpanda profile.
  • S3 sink — format and partitioning parity with the Kafka Connect S3 sink, and the AWS default credential chain.

Notable features​

  • Protocol v3 and lossless structured values. int64, uint64, bytes, and decimal now cross the plugin boundary exactly — no 2^53 rounding, no []byte→base64, no NaN/Inf→string.
  • Protobuf decode for Confluent Schema Registry (PB-1 to PB-6).
  • Secrets at rest. ${VAR} references are no longer stored in plaintext; they are resolved when a connector, processor, or DLQ is provisioned.
  • Processor batch API and a stateless declaration, plus processor input isolation that keeps DLQ records faithful.
  • Connector observability. Per-connector record and error counters.
  • Python connector SDK GA. One merged conduit.connector / conduit.client distribution, and conduit connector new --lang python.
  • Catalog. Public quality tiers and a license scorecard.
  • Engine. arch-v2 grows toward the single engine, and the benchi regression gate is live.

Migrating​

Start with conduit migrate kafka-connect, or read the migration command's --help. The command prints a compatibility report for every run, so you can see exactly what translated, what was approximated, and what would block before you switch.

scarf pixel conduit-site-changelog