I have a set ok kotlin data classes implementing sealed classes and containing…
- User
I have a set ok kotlin data classes implementing sealed classes and containing primitives or other classes. How do I automatically transform them to Protos file for protobuf? I need to share the Protos files with backend
- Assistant
Okay, this is a common desire, but unfortunately, there's no out-of-the-box, universally adopted, "magic" tool that perfectly and automatically converts arbitrary Kotlin data/sealed classes directly into
.protofiles with 100% fidelity and without some setup or custom logic.The main challenge is that while there's overlap, Kotlin's type system and Protobuf's type system have differences, and decisions need to be made (e.g., how to number fields, package names, exact type mappings).
Here are the primary approaches, from most manual to most automated (and complex to implement):
1. Manual Conversion (with Conventions)
-
How it works: You manually write the
.protofile, keeping it in sync with your Kotlin classes. -
Pros: Simple to understand, full control.
-
Cons: Error-prone, tedious, not automatic. Becomes unmanageable for large schemas.
-
Best for: Very small projects or initial prototyping.
-
To make it manageable:
- Establish strict conventions for field numbering (e.g., alphabetical, or add comments in Kotlin with desired numbers).
- Keep Kotlin and Proto side-by-side during development.
// Kotlin package com.example.models sealed interface ApiResponse { // @ProtoField(1) // imaginary annotation for numbering data class Success( // @ProtoField(1) val data: String, // @ProtoField(2) val items: List<Item> ) : ApiResponse // @ProtoField(2) data class Error( // @ProtoField(1) val code: Int, // @ProtoField(2) val message: String ) : ApiResponse } data class Item( // @ProtoField(1) val id: Long, // @ProtoField(2) val name: String ) enum class Status { PENDING, // @ProtoValue(0) ACTIVE, // @ProtoValue(1) INACTIVE // @ProtoValue(2) }// Corresponding Proto syntax = "proto3"; package com.example.models; message ApiResponse { oneof result { Success success = 1; Error error = 2; } } message Success { string data = 1; repeated Item items = 2; } message Error { int32 code = 1; string message = 2; } message Item { int64 id = 1; string name = 2; } enum Status { STATUS_UNSPECIFIED = 0; // Or PENDING = 0; PENDING = 0; // if you want to match Kotlin exactly, start from 0 ACTIVE = 1; INACTIVE = 2; }
2. Custom Kotlin Script using Reflection (Limited)
- How it works: Write a Kotlin script that uses reflection to inspect your compiled Kotlin classes and generate a
.protostring. - Pros: Can be somewhat automated.
- Cons:
- Reflection loses some compile-time information (e.g., original property order for field numbering can be unreliable).
- Handling all edge cases (nullability, generics, complex types, imports) is hard.
- Field numbering strategy is crucial and hard to infer reliably without annotations.
- Typically run as a separate step.
- Best for: Simple, flat data structures where you can make assumptions.
3. Custom KSP (Kotlin Symbol Processing) Processor (Most Powerful & Complex)
-
How it works: This is the most robust way to achieve true automation from Kotlin source. You write a KSP processor that analyzes your Kotlin source code at compile time and generates the
.protofiles. -
Pros:
- Access to rich type information directly from the Kotlin compiler.
- Can be highly configurable with custom annotations (e.g.,
@ProtoMessage,@ProtoField(number = 1),@ProtoOneOf). - Integrates directly into the build process.
- Type-safe.
-
Cons:
- Significant development effort to write a KSP processor. This is not a trivial task.
- Requires understanding of KSP APIs and compiler internals.
-
Best for: Larger projects where the Kotlin classes are the "source of truth" and you need reliable, automated generation.
Conceptual KSP Processor Outline:
-
Define Annotations (Optional but Recommended):
@Target(AnnotationTarget.CLASS) annotation class GenerateProtoMessage(val protoPackage: String = "") @Target(AnnotationTarget.PROPERTY) annotation class ProtoField(val number: Int, val customName: String = "") @Target(AnnotationTarget.CLASS) // For sealed classes annotation class GenerateProtoOneOf @Target(AnnotationTarget.CLASS) // For enums annotation class GenerateProtoEnum -
Implement
SymbolProcessor:- Find classes annotated with
@GenerateProtoMessage/@GenerateProtoEnum. - For each class:
- Determine the Protobuf package (from annotation or Kotlin package).
- Generate the
message MessageName { ... }orenum EnumName { ... }string. - Iterate through properties:
- Get the property name and type.
- Use
@ProtoFieldto get the field number and custom name. If no annotation, devise a strategy (e.g., alphabetical order for names, assign numbers sequentially - CAUTION: this is brittle if you reorder/rename properties without updating numbers). - Map Kotlin types to Protobuf types:
String->stringInt->int32Long->int64Boolean->boolFloat->floatDouble->doubleByteArray->bytesList<T>->repeated TProtoMap<K, V>->map<KProto, VProto>- Nullable types (
String?) ->optional string(or juststringin proto3, as all non-repeated scalar fields are optional by default, message fields are implicitly optional). - Other data classes -> reference to another message (ensure it's also processed or imported).
- Enums -> reference to a proto enum.
- For
sealed classes/interfaces annotated with@GenerateProtoOneOf:- Iterate through direct inheritors.
- Generate a
oneofblock. Each field in theoneofwill be one of the inheritors. oneof oneof_name { Type1 field1_name = number1; Type2 field2_name = number2; }
- Use KSP's
CodeGeneratorto write the.protofile to a directory (e.g.,build/generated/ksp/main/proto/).
- Find classes annotated with
-
Build Integration:
- Add KSP plugin to your
build.gradle.kts. - Add your processor as a
ksp(...)dependency. - Configure the
protobuf-gradle-pluginto pick up generated.protofiles from the KSP output directory if you also need to generate code from these protos (e.g., for gRPC or backend usage).
- Add KSP plugin to your
-
4. Using an Intermediate Format (e.g., JSON Schema)
- How it works:
- Generate JSON Schema from your Kotlin classes (e.g., using
kotlinx.serialization.schemaexperimental feature or third-party libraries likeKSON). - Use a tool to convert JSON Schema to Protobuf (e.g.,
json-schema-to-protobuf).
- Generate JSON Schema from your Kotlin classes (e.g., using
- Pros: Leverages existing tools for parts of the process.
- Cons:
- Potentially lossy conversion (JSON Schema and Protobuf have different feature sets).
oneoffor sealed classes might not translate perfectly without custom handling.- Adds an extra step and dependency.
kotlinx.serialization.schemais experimental.
Recommendation:
- For small projects or if you're just starting: Start with Manual Conversion to understand the mapping. Define clear conventions for field numbering.
- For robust, long-term automation where Kotlin is the source of truth: Invest in a Custom KSP Processor. This is the most "correct" and integrated way, but it's a project in itself.
- Start simple: handle basic data classes and primitives.
- Incrementally add support for lists, maps, enums, nested classes, and finally sealed classes/interfaces as
oneof. - Crucially, decide on a field numbering strategy. Using an explicit
@ProtoField(number = ...)annotation on properties is the most robust way to prevent accidental breaking changes. If you rely on order or naming, you're fragile.
- Avoid reflection-based scripts for anything serious due to their limitations and brittleness.
Key considerations for any automated solution:
- Field Numbering: This is CRITICAL for Protobuf. Field numbers must be unique within a message and should not change once in use. An explicit annotation (
@ProtoField(1)) is the best way. - Package Names: How will
package your.kotlin.package;map topackage your.proto.package;in the.protofile? - Type Mapping: Ensure you have a clear mapping for all Kotlin types you use.
- Default Values: Protobuf has specific rules for default values.
- Nullability/Optionality: In proto3, scalar fields don't distinguish between "not set" and "set to default value" unless you use
optionalkeyword or wrapper types. Message fields are always "optional" in behavior. - Imports: If your Kotlin classes reference types from other modules/packages, your generated
.protofiles will need correspondingimportstatements. - Naming Conventions: Kotlin
camelCasevs. Protobufsnake_casefor field names (though Protobuf compilers often handle this, it's good to be consistent).
Building a KSP processor is the most powerful path but also the most involved. If your schema is relatively stable and not too large, a disciplined manual approach might suffice, or a very targeted script for your specific class patterns.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.