KiwiCodec

KiwiCodec is a pure Elixir implementation of the Kiwi schema binary codec: a compact, schema-driven message format similar in spirit to Protocol Buffers but with its own wire encoding.

The package stays generic: product-specific .kiwi schemas can live in companion packages.

Scope

KiwiCodec owns:

• Kiwi wire primitives: varuint, zigzag int, uint64/int64, varfloat, null-terminated strings, and length-prefixed byte arrays
• .kiwi schema parsing and validation
• Schema interpretation for parsed schemas
• Field metadata, including original schema field names
• An idiomatic Elixir DSL and code generator for static modules
• Generic chunk container helpers

Static modules

Generated modules are regular Elixir structs:

defmodule Example.Node do
  use KiwiCodec, kind: :message

  field :id, 1, type: :uint
  field :name, 2, type: :string
end

node = %Example.Node{id: 42, name: "hello"}
binary = KiwiCodec.encode(node)
node = KiwiCodec.decode(binary, Example.Node)

Enums use atoms:

defmodule Example.Kind do
  use KiwiCodec, kind: :enum

  enum_value :none, 0
  enum_value :frame, 4
end

Schema compilation

For application code, compile schema text into Elixir modules and use the static struct API:

mix kiwi.gen schema.kiwi --module-prefix MyApp.Schema --out lib/generated

For tests and tooling, modules can also be compiled in memory:

KiwiCodec.compile_schema!(schema_text, module_prefix: MyApp.Schema)

KiwiCodec.parse_schema!/1 only parses schema text into an AST. It does not create modules by itself.

Schema interpretation

When a schema is loaded at runtime and you do not want to generate modules, use KiwiCodec.SchemaInterpreter:

schema = KiwiCodec.parse_schema!(schema_text)
binary = KiwiCodec.SchemaInterpreter.encode(schema, "Thing", %{"id" => 1, "name" => "demo"})
value = KiwiCodec.SchemaInterpreter.decode(schema, "Thing", binary)

Transform modules

Modules can override transform_module/0 for custom normalization before encode and after decode:

defmodule Example.Transform do
  @behaviour KiwiCodec.TransformModule

  def encode(message, _module), do: message
  def decode(message, _module), do: message
end

Binary schemas

binary_schema = KiwiCodec.Schema.Binary.encode(schema)
schema = KiwiCodec.Schema.Binary.decode(binary_schema)

Rustler decoder generation

KiwiCodec.RustlerGenerator.source/2 returns complete generated Rust source for native Rustler decoders. Use it from rustq.exs and let mix rustq.gen own writing and freshness checks:

use RustQ.Config

schema = KiwiCodec.parse_schema!(File.read!("priv/schema.kiwi"))

generate :native_decoders, "native/my_nif/src/generated.rs" do
  content KiwiCodec.RustlerGenerator.source(schema,
    entrypoints: ["Node"],
    module_prefix: "MyApp.Schema"
  )
end

Definition names infer NIF names such as decode_node. For Rustler projects, prefer keeping the native API list in one non-Rustler metadata module:

defmodule MyApp.Native.Nifs do
  @moduledoc "Native NIF stubs exposed by MyApp."

  @stubs [decode_node: 1, decode_image: 1]

  def stubs, do: @stubs
end

Then use the same metadata from rustq.exs:

generate :native_decoders, "native/my_nif/src/generated.rs" do
  content KiwiCodec.RustlerGenerator.source(schema,
    entrypoints: {:nif_stubs, MyApp.Native.Nifs},
    module_prefix: "MyApp.Schema"
  )
end

The generated file includes Rustler imports, RustQ-provided atom/struct helpers, schema decoders, and requested NIF entrypoints. The native crate must provide the Kiwi Decoder type used by the generated code; by default it is imported from crate::runtime::Decoder, or pass decoder: "some::path::Decoder".

When generating skip decoders, pass the Rust source file that defines that Decoder with decoder_sources:. KiwiCodec then authors the shared skip value helpers with RustQ defrust, and RustQ reads the real decoder method signatures to infer ? propagation:

content KiwiCodec.RustlerGenerator.source(schema,
  features: [:full, :sparse, :skip],
  entrypoints: {:nif_stubs, MyApp.Native.Nifs},
  module_prefix: "MyApp.Schema",
  decoder_sources: ["native/my_nif/src/runtime.rs"]
)

The option is only used for :skip generation. Full and sparse decoders keep the compact default helper path when :skip is not requested.

For generator internals, especially the compact Rust macro boundary used to keep large schemas readable and small, see the Rustler generator architecture guide.

Containers

binary = KiwiCodec.Container.build([KiwiCodec.Container.deflate("schema"), KiwiCodec.Container.deflate("data")])
parsed = KiwiCodec.Container.parse(binary)

Benchmarks

elixir bench/codec_bench.exs

Development

mix deps.get
mix ci

Installation

Once published, add it to your dependencies:

def deps do
  [
    {:kiwi_codec, "~> 0.2.2"}
  ]
end