3D painting: full surface unique texturing

One of my favorite pet ideas which I've prototyped and not quite put into a production pipeline is 3D Painting, which is what I'm calling a set of techniques for acheiving full unique surface texturing without having to store masses of data on disk.

Instead, I extend and generalize the deferred shading concept to include more flexible rendering primitives that can update the G-buffer. Think of the shade tree or expression tree of a shader, and how that can be broken up with deferred shading. Instead of evaluating it all in one go as in forward shading, you break up into stages. In the first stage (deferring), you populate the original G-buffer with some geometric and material properties. Typically, the next stage is then to render in light volumes which compute the BRDF based on G-buffer samples and accumulate illumination. But why stop there? You can also add in more complex procedural effects that happen before lights but after the G-buffer, such as procedural texturing primitives and decals. Really you can break up any complex procedural shader into a series of substeps which can be spatially limited to effect certian volumes.

Lights rendered as 3D volumes are an example of one G-buffer modifier, but another one is a 3D volume that 'splats' a texture layer or 'procedural' texture effect into the G-buffer, modifying just the albedo (or normal, or whatever). So, with the system, the idea is that artists can paint in directly into the scene (stored currently as boxes with alpha textures projected onto them), painting in layered effects which will touch all pixels in that volume. I've combined this was a 3D projected texture blending technique that even avoids the need to UV-map the surface! This is super cool for terrains! No seams! No stretching on vertical surfaces! And best of all, you can paint onto any general meshes sitting on the terrain - so your artists can throw some untextured rock models in and paint them with the exact same tools they use to paint the terrain.

Of course, it can also be used for decals, 3D destruction texturing for buildings, and even for dirt and grime layers for buildings and your general world. After coming up with this in my ivory tower so to speak, I found out that some of our other teams were already experiementing with vaguely similar 2D decal based techniques for multi-layer texturing effects - to great effect. What I'm doing differently then is extending it to 3D volumes which gets around all kinds of issues and allows it to work on organic surfaces, and 3D projective texturing, which magically solves all kinds of organic surface texturing problems. But the aligned 2D quad case is something of a simplification which is still useful for buildings which are all right angles.

The disadvantage, compared to the explicit virtual texture route, is a memory-performance tradeoff as you need to composite all the different painting layers in real-time on the GPU. A couple of simple texture decals is no problem, but 3D projective texturing requires looking up each texture 3 times, and the basic layer 'brushes' I'm using typically have 4 texture layers, AND you really need some anisotropic filtering for it too look good, so this can quickly get out of hand. Fortunately, the G-buffer reprojection cache is there to save the day.

Another possible optimization is to composite G-buffer blending ops in the reverse of the typical order, front-to-back or last-to-first in this case. The alpha blending function is reversible, with a little extra storage. This would allow population of a stencil mask or hi-z to kill later (earlier) decals/edits which won't contribute to the final g-buffer as later operations are known to completely blend over them. Haven't tried that yet.