Visual Blocks
Unleash your creativity
Visual Blocks for ML is a Google visual programming framework that lets you create ML pipelines in a no-code graph editor. You – and your users – can quickly prototype workflows by connecting drag-and-drop ML components, including models, user inputs, processors, and visualizations.
Try Visual BlocksTry our Visual Blocks Colab integration examples: Cartoonization, Style Transfer
Learn Visual Blocks
Tutorials
Create Effects
Use a live camera and ML models to create effects like face stickers.
Compare Models
Import custom ML models and compare side-by-side results.
Integrate with Colab
Use Visual Blocks in Colab and create nodes for custom Python code.
Publications
Our paper describes the iterative design process of an earlier version of Visual Blocks for ML and how ML practitioners could accelerate their workflow, make more informed decisions, analyze strengths and weaknesses of ML models, and holistically evaluate model behavior with real-world input.
FAQ
What is Visual Blocks?
Visual Blocks is a visual programming framework for rapidly prototyping and experimenting with ML models and pipelines. Using Visual Blocks, you can mix and match premade ML models and pipeline components in a graphical editor. The editor renders your pipeline as an interactive node graph where you can connect components by dragging and dropping nodes.
Visual Blocks provides three core components:
- A library of nodes representing server-side models, in-browser models, and pre- and post-processing visualizers.
- A runner that runs the graph and displays the output of all or specific nodes.
- A visual editor where you can create your graph by connecting nodes with matching inputs and outputs.
Why did you create Visual Blocks?
Recent advances in ML have enabled high-performance, real-time solutions to multimedia problems such as human body segmentation for video, depth estimation for 3D reconstruction, audio processing for remote communication (coming soon), and hand and body tracking.
However, iterating on ML-based multimedia prototypes can be challenging and costly. It often requires a cross-functional team of ML practitioners to fine-tune models, evaluate robustness, identify strengths and weaknesses, measure performance, and develop applications. Because models have to be repeatedly updated and integrated, the workflow is ill-suited to prototyping and experimenting.
Visual Blocks is designed to address these challenges. Its visual editor provides a no-code development environment where you can create and connect different components to rapidly build an ML pipeline and see the results in real time. Visual Blocks has the following goals:
- Lower development barriers for ML-based real-time multimedia applications.
- Empower users to experiment without worrying about coding or technical details.
- Provide premade models and datasets for common tasks such as body segmentation, landmark detection, and portrait depth estimation.
- Support various types of input data (image, video, audio (coming soon)) and output modalities (graphics, sound).
- Provide interactive data augmentation and manipulation with drag-and-drop operations and parameter sliders.
- Enable side-by-side comparisons of models and outputs at different stages of the pipeline.
- Make it easy to share visualizations and publish multimedia pipelines to the web.
- Facilitate collaboration between designers and developers.
- Provide a common language for describing ML pipelines.
Ultimately, we hope Visual Blocks will help you accelerate your workflow, make more informed decisions about model selection and tuning, analyze the strengths and weaknesses of different models, evaluate model behavior with real-world input, and share your applications with the world.
How does Visual Blocks work?
Visual Blocks is mainly written in JavaScript. It leverages TensorFlow.js and TensorFlow Lite for ML capabilities and three.js for graphics rendering.
The Visual Blocks interface helps you rapidly build and interact with ML models using three coordinated views:
- A node library containing more than 30 nodes (including nodes for image processing, body segmentation, and image comparison) and a search bar for filtering.
- A node-graph editor that lets you create multimedia pipelines by dragging and dropping nodes from the library.
- A preview panel that shows pipeline inputs and outputs, provides immediate updates, and enables comparison of different models.
How do I get started?
Currently, Visual Blocks is available as a Colab integration. The integration lets you build interactive demos in Colab notebooks using prebuilt components.
To get started with Visual Blocks, try the demos or one of the following Colab examples:
- Visual Blocks Quick Start for Colab: Cartoonization
- Visual Blocks Quick Start for Colab: Style Transfer
Then check out the documentation at the Visual Blocks GitHub repository.
Visual Blocks for ML is intended to be used in line with Google’s AI principles.
Where can I learn more?
To learn more about Visual Blocks, see the following resources:
