Transform Dependencies
In this example, we will show how import rust crates otherwise not included. In this example, we will import the regex library and verify that the source topic reads proper https website links.
Prerequisites
This guide uses local
Fluvio cluster. If you need to install it, please follow the instructions at here.
Transformation
Below is the syntax for importing multiple crates.
transforms:
- operator: (transfrom operator)
dependencies:
- name: (library a)
version: (version number)
- name: (library b)
version: (version number)
(... more crates ...)
run: |
(...)
In our example, the following is our transfrom operator.
transforms:
- operator: filter
dependencies:
- name: regex
version: "1"
run: |
fn verify(input: String) -> Result<bool> {
let re = regex::Regex::new(r"^(https?:\/\/)?([a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(\/[^\s]*)?$")
.unwrap();
Ok(re.is_match(&input))
}
Running the Example
Copy and paste following config and save it as dataflow.yaml
.
# dataflow.yaml
apiVersion: 0.5.0
meta:
name: regex-example
version: 0.1.0
namespace: examples
config:
converter: raw
topics:
sentences:
schema:
value:
type: string
validlink:
schema:
value:
type: string
services:
verify-service:
sources:
- type: topic
id: sentences
transforms:
- operator: filter
dependencies:
- name: regex
version: "1"
run: |
fn verify(input: String) -> Result<bool> {
let re = regex::Regex::new(r"^(https?:\/\/)?([a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(\/[^\s]*)?$")
.unwrap();
Ok(re.is_match(&input))
}
sinks:
- type: topic
id: validlink
To run example:
$ sdf run
Produce sentences to in sentence
topic:
$ echo "hello world" | fluvio produce sentences
$ echo "https://www.fluvio.io" | fluvio produce sentences
Consume topic validlink
to retrieve the result in another terminal:
$ fluvio consume validlink -Bd
https://www.fluvio.io
Cleanup
Exit sdf
terminal and clean-up. The --force
flag removes the topics:
$ sdf clean --force
Conclusion
We just covered another basic operator in SDF, the Flat-Map Operator. The Flat-Map is a powerful operator. As a matter of fact, the Flat-Map operator can be used inplace of other operators.