Note: I wrote this post back in the day I worked as an AI Engineer at The Grid. It was a crazy ride, with lots of interesting people and challenges. I left The Grid years ago but some ideas are still relevant to augment design. The content of this post were originally published on The Grid itself and since there's no mirrors around, I'm sharing it here and I hope you enjoy it.
Augmenting and automating design of websites involve many steps and the areas I had my hands allover was on Computer Vision (CV) and Generative Art. We used a mix of traditional CV, generative algorithms and ML models, that when applied to design, got us the following features:
- Saliency extraction: crop to region, text placement
- Face detection: emotion recognition, detect salient regions
- Delaunay triangulation: generative backgrounds, picture styling
- Image classification: extract information from pictures
- Sentiment analysis: extract general mood of a picture
- Color extraction: extract/quantize colors from pictures
Let's take a look on some of them in more details, shall we?
When we look into a picture we can naturally determine what points get our attention first. A human face, a specific object, a region with higher contrast, those are both examples of salient areas. We can easily recognize those areas, but computers don't. So we have to teach them.
The Grid used a saliency detection algorithm which analyzes a given image looking for regions of high contrast. In summary, it divides the original image by means of little segments. Those segments are analyzed and just the most contrasting ones are keep. We call them salient regions (as shown in the side figure).
Knowing which regions are most salient we can now crop the original image to show only those regions — to fit on any screen size, for example. Or we can place text on top of images avoiding the salient regions.
Generative images and image processing
Besides analytics, we can use extracted information from text, images or videos to synthesize new content. We can generate an infinite number of unique images using combinations of different shapes, curves and colors.
We can apply more complex algorithms like Delaunay Triangulation to obtain meshes for favbanners or backgrounds.
Or we can go beyond and use the same algorithm to style existing pictures.
Using our open-source image processing pipeline imgflo we can manipulate images. It is possible to create any image filter and apply them to enhance pictures.
Imgflo is a runtime for Flowhub, a visual programming environment that made it simple to build image processing graphs.
It's more than 6 years since I wrote this post. Deep Learning happened, Computer Vision was completely changed by it and other ML methods. However, the fundamental problems on design are still there and we can leverage a lot from ML/CV to augment and automate graphical and web design.
If you want to know more about my last developments on blending AI/ML with design and creativity, follow me on Twitter!