<aside> 🚧

This is a living document, and a work in progress. You’re going to see missing or incomplete pieces. It’s meant to be an easier way to progressively build the documentation for adding modules support to an SDK, until it’s ready to be committed and published in the dagger/dagger repo.

</aside>

<aside> <img src="/icons/info-alternate_blue.svg" alt="/icons/info-alternate_blue.svg" width="40px" />

You may see code samples written in Python, but they are meant to be pseudo-code.

</aside>

Overview

There’s two main parts for adding modules support:

1. Create an executable (aka, runtime entrypoint) that knows how to read and execute user code
2. Create a Dagger module, using Go, to interact with the Dagger engine

Technical decisions

There’s things that you need to figure out technically, which can be done before even starting with the runtime module. You’ll need to be able to:

• Figure out the DX that users need to use to define their objects, functions, etc, and possibly associated metadata
• Parse the user code and extract all the information needed to report what the module exposes to the Dagger API
• Call a user function based on the deserialized input
  • Serialize and validate inputs from JSON
  • Find and execute function based on the parent object’s name and function name
  • Validate and deserialize the function’s output to JSON

Additionally, parsing the user’s code can be done either at:

• run time - all non-Go modules do this currently
  • This means the code is parsed dynamically every time a function is executed
  • The Python SDK is a simple example to learn from, using this method
• codegen time - only Go does this currently
  • This means the code is parsed only during codegen, where the needed function calls are generated into the code that will later be executed by the runtime entrypoint
  • You can see an example of this method with any Go module, just open the generated dagger.gen.go file

The choice of parsing the code during run time or codegen time is a matter of performance, but also productivity. Python has the types readily available at runtime so there’s no noticeable overhead. Go, on the other hand, is a compiled language and uses lower level AST parsing that’s more wasteful to do every time. However, with the codegen path, you may get more cache invalidation during the codegen phase because any change in function signatures generates a different output. The code may also be harder to maintain. It pays to measure which approach is more worth it if not clear.

Example

Consider the following function, defined in a user module:

class MyModule:
		def echo(self, msg: str) -> str:
				return msg

To start, you can mock the API calls that will get the necessary inputs, and save the right output:

# Name of the object
parent_name = "MyModule"

# State of the object instance
parent = {}

# Name of the function to call
name = "echo"

# Function arguments
input_args = {
		"msg": "Hello!",
}

# Abstraction for finding and executing function
result = get_result(parent_name, parent, name, input_args)

assert result == "Hello!"