Writing Custom Functions in Rust

[depatchedmode]

Feature Description

As we solicit feedback from folks, they need to be able to write custom functions in order to:

1. Evaluate and understand the capabilities of the Everywhere Computer
2. Imagine clear use cases within their business

Use Cases

• Creating bespoke workflows for your own problem domain
• Really running EC through the trials
• Writing a basic "Hello World" rust module and seeing how it runs as a workflow

Signals of Success

More function diversity in each run, and overall.

Current Workarounds (optional)

Technically possible write now, but requires making changes to core code. See Philipp's tear down of how it's done presently: https://www.loom.com/share/19c4651b5ae6419ea091cd6c2fcb049f?sid=8739f44e-e88e-4a29-9446-5eb9a39d59aa

Summary of current process is:

1. in your local Homestar repo:

• write a function in homestar-functions/src/lib.rs
• add type definitions to 'homestar-functions/wit/host.wit`
• compile by running cargo build -p homestar-functions-test --target wasm32-unkown-unknown --profile release-wasm-fn
• now turn it into a Wasm component: wasm-tools component new ... -o functions.wasm
• ingest into local IPFS node

2. in your local Control Panel repo:

• launch the control panel npm run dev
• make function available in dropdown. update the VITE_WORKFLOW_RESOURCE with the newly created module's CID in .env
• add the function signature to the const FUNCTION_NAMES array in NewFunctionNode.svelte
• in function-template.ts:
  • rewrite generateFunction as so:

     // philipp's snippet goes here
    

  • specify default parameters in the const DEFAULT_PARAMS object
• in graph.ts add function definitions to const FUNCTION_PARAMS

Extreme MVP is to document the workaround of manually writing and ingesting wasm modules.
Consider this a map of the work to be done in pursuit of a cleaner DX for authoring -> publishing in your local registry of functions.

Evidence of Value (Optional)

Table stakes! You can't have a computer if there's nothing to compute.

Tasks & Jobs Stories

• Decide where to store function configuration — in the getting-started repo?
  • Make a decision on whether to change that repo's name everywhere-computer/getting-started#3
• Control Panel: generalize function invocation
• Control Panel: move parameter editing in visual workflow builder to its own pane, to avoid the need for different node sizes based on
• Control Panel: display correct parameter inputs based on parameter type definition
• Control Panel: support for multiple modules
• Control Panel: ingestion of modules and functions via UI (drag n' drop, file picker)
• Guide: Writing Custom Functions docs#4
• Control Panel: Track the usage of custom functions

Additional Context (Optional)

Boris is going to start by dog fooding the process.

[matheus23]

Haha this so very closely re-created my "script" I wrote for the video, so here's my script for reference:

---

Custom Homestar Function Script

• setup

  • ipfs --repo-dir ... --offline daemon
  • run homestar
  • control panel: npm run dev

• Homestar repo, homestar-functions folder

  • Add function
  • Add interface under host.wit
  • Compile using --target wasm32... --profile release-wasm-fn
  • Use wasm-tools component new ... -o ...
  • Add to ipfs via ipfs --repo-dir .ipfs add functions.wasm --cid-version 1

• control-panel repo

  • Change .env
  • Change function-template.ts: Use HSWorkflow.invocation
  • Change NewFunctionNode.svelte: Add to FUNCTION_NAMES
  • Change graph.ts: Add params.

    fn unsafe_encrypt(data: Vec<u8>, key: u32) -> Vec<u8> {
        let loaded = image::load_from_memory(&data).unwrap();
        let stream_bytes = loaded.width() * loaded.height() * 4;
        let mut output = blake3::Hasher::new_derive_key(
            "poor man's stream cipher. Please use a proper AEAD with a proper password kdf instead.",
        )
        .update(&key.to_le_bytes())
        .finalize_xof();

        let mut key_stream = vec![0u8; stream_bytes as usize];
        output.fill(&mut key_stream);

        let mut rgba_image = loaded.into_rgba8();

        for (i, pixel) in rgba_image.pixels_mut().enumerate() {
            let offset = i * 4;
            pixel.0[0] ^= key_stream[offset];
            pixel.0[1] ^= key_stream[offset + 1];
            pixel.0[2] ^= key_stream[offset + 2];
            pixel.0[3] ^= key_stream[offset + 3];
        }

        let mut buffer: Vec<u8> = Vec::new();
        rgba_image
            .write_to(&mut Cursor::new(&mut buffer), image::ImageOutputFormat::Png)
            .unwrap();
        buffer
    }

    fn unsafe_encrypt_base64(data: String, key: u32) -> Vec<u8> {
        let base64_encoded_png = data.replace("data:image/png;base64,", "");
        let decoded = general_purpose::STANDARD
            .decode(base64_encoded_png)
            .unwrap();
        Self::unsafe_encrypt(decoded, key)
    }

  const homestarFunctionName = `${functionName}${base64 ? '-base64' : ''}`

  const inv = {
    name: label,
    resource: import.meta.env.VITE_WORKFLOW_RESOURCE,
    ...(needs ? { needs } : {}),
    args: Object.values({
      data,
      ...(args ? args : DEFAULT_PARAMS[functionName])
    }),
    func: homestarFunctionName
  };

  return HSWorkflow.invocation(inv)

[depatchedmode]

Discussion to start: where should we deposit function/module configuration?

My proposal would be to rename the getting-started repo to default-workspace and have both homestar and the control panel be pointed to config/definitions that life there. That can be the source for both custom local functions, and things pulled down from any other registries.

Issue where I propose the rename is here https://github.com/everywhere-computer/getting-started/issues/3

Thoughts?

[bmann]

This is roughly "here's what I want to run on my local node / network of nodes", yes?

So this would in part be booting up your homestar node and pointing at this set of configs, expanded to include "on startup, always cache / load these functions"

So, it would either be a list of functions by content address or their equivalent DNS link, or local file paths, which then get uploaded / cached by the local node.

[depatchedmode]

Yep, exactly. And when we have a CLI that boots up homestar, control panel, etc it'd know about that workspace and pass along important bits from it to the initialization of any other processes.