Pipeline Manager

Copyright (c) 2022-2024 Antmicro

Pipeline Manager is a data-based, application-agnostic web application for creating, visualizing and managing dataflows in various applications. It does not assume any properties of the application it is working with, thanks to which fast integration with various formats is possible.

Pipeline Manager documentation | Demo application

It provides functionality for:

• visualizing and editing dataflows,
• saving and loading dataflows,
• communicating with an external application to delegate advanced validation, execution of the defined graph and conversions to and from native formats of the graphs.

Pipeline Manager aims to simplify the process of developing graph-based graphical interfaces for applications that are modular and have a graph-like nature.

Prerequisites

Pipeline Manager requires npm, python and pip for installing dependencies, building and (optionally) running - use package manager to install those packages.

The backend of the application has a list of requirements in the pyproject.toml file. They can be installed using pip:

pip install .

All of npm modules needed for the frontend of the application are installed automatically during build. They can be found in the ./pipeline_manager/frontend/node_modules directory after the application is built.

Building and running

Pipeline Manager can be built in two different ways as:

• static HTML application, that can be run in a browser, without any additional backend server,
• regular web application that is designated to communicate and cooperate with an external application (like Kenning).

Static HTML application

To build Pipeline Manager as a static HTML application, in the root directory run:

./build static-html

For available flags, run:

./build static-html -h

To run the built application, open ./pipeline_manager/frontend/dist/index.html in a preferred browser. As an example, if the browser of your choice is firefox you should run:

firefox ./pipeline_manager/frontend/dist/index.html

After running Pipeline Manager you can use sample specification under ./examples/sample_specification.json to check the visualization and editing of pipelines. Additionally, ./examples/sample_dataflow.json can be used to see how dataflows are stored.

The specification can be loaded in the webpage using the Load specification option in the main menu. The dataflow can be loaded in the webpage using the Load graph file option in the main menu.

What is more, the specification and the dataflow can be provided as URL arguments:

• spec - should contain URL to the specification file,
• graph - should contain URL to the dataflow file.

For example:

firefox "./pipeline_manager/frontend/dist/index.html?spec=https%3A%2F%2Fraw.githubusercontent.com%2Fantmicro%2Fkenning-pipeline-manager%2Fmain%2Fexamples%2Fsample-specification.json&graph=https%3A%2F%2Fraw.githubusercontent.com%2Fantmicro%2Fkenning-pipeline-manager%2Fmain%2Fexamples%2Fsample-dataflow.json"

Will fetch and use specification and dataflow from the GitHub repository for this project. The URLs need to be encoded.

It is possible to add a default specification JSON to the generated HTML. It just needs to be provided as the second argument of the ./build script:

./build static-html <path-to-specification-json>

It is also possible to add a default dataflow that will be loaded on the start of the application, e.g.:

./build static-html <path-to-specification> <path-to-dataflow>

To be able to use some additional assets, like icons for nodes, run:

./build --assets-directory <path-to-assets-dir> static-html <path-to-specification> <path-to-dataflow>

To change the title of the editor and page, use --editor-title flag, e.g.:

./build --editor-title 'Graph editor' static-html <path-to-specification> <path-to-dataflow>

For details on how to write specification, check:

• Pipeline Manager documentation
• Specification format
• Dataflow format
• Examples in examples/ directory - in the directory you can find sample specification files (with -specification.json suffix), usually paired with supported dataflow files (with -dataflow.json suffix)

For example, run:

./build static-html ./examples/sample-specification.json ./examples/sample-dataflow.json --output-directory ./pipeline-manager-demo

After successful build, run:

firefox ./pipeline-manager-demo/index.html

You should get a graph view similar to the one in the documentation's demo.

Web application

To build Pipeline Manager to work with an external application (like Kenning), in the root directory run:

./build server-app

For available flags, check:

./build server-app -h

In this scenario, the backend server is expected to serve the Pipeline Manager content. To do that, in the root directory run:

./run

By default, the backend server runs on http://127.0.0.1:5000. In addition to using the sample specification you can also connect the third-party application (e.g. Kenning), edit its pipeline, validate it and run it.

Miscellaneous

Development

To run a development server which automatically recompiles the project after detecting any changes, in ./pipeline_manager/frontend directory run:

npm run serve-static

in case of a static mode, and run:

npm run serve

in case of a regular web application mode.

Validation

Pipeline Manager also includes a validation tool you can use during specification and dataflow development.

To validate an existing specification, run the following in the root directory:

./validate <specification-path>

To validate an existing specification and dataflow, run the following in the root directory:

./validate <specification-path> <dataflow-path>

Replace both specification-path and dataflow-path with the actual paths to the JSON configuration file you want to validate. When running the validation tool for the first time, make sure to include the --instal-dependencies flag.

Cleanup

To remove the installed npm dependencies and the built application, run the following in the root directory:

./cleanup

Using Pipeline Manager as a Python module

To install Pipeline Manager with pip, run:

pip install -U git+https://github.com/antmicro/kenning-pipeline-manager.git

To work directly with the repository, install the module with:

pip install -e .

All Pipeline Manager scripts can then be used from the command-line interface:

pipeline_manager
usage: pipeline_manager {build,run,validate,cleanup}