Enhanced shader syntax

As explained in A high level shader construction syntax – Part I the proposed shader syntax is an extension over SM4 and SM5. It is simple enough to be parsed with custom regex-based code.

One of the requirements I had when designing this is that vanilla HLSL shader code going through the translator should remain unchanged.

Usually a shader is a mini-pipeline with predefined steps that only vary slightly. Let’s take for instance the pixel shader that populates a GBuffer with per-pixel depth and normal. It has three distinct steps – take the depth of the pixel, take the normal of the pixel and output both. Now here comes the branching per-material, some materials might have normal maps while others might use the interpolated normals from the vertices. However the shader just has to complete these 3 steps – there is no difference how you get the normal.

Nearly all shader code can be simplified as such simple steps. So here I came up with the idea of what I called ‘polymorphics‘. They are placeholders for functions that perform a specific operation (i.e. fetch normal) and can be varied per-material.

The code for a simple GBuffer pixel shader could look like this:

The keyword ‘pixel_shader‘ is required so that the translator knows the type of function it is working on. We have two declared polymorphics –  MakeDepth and GetWorldNormal with the functions (called ‘atoms‘) that can substitute them.

If the material has a normal map, after the translation process this shader looks like this:

There is much more code generated by the translator – the polymorphics have been substituted by ‘atoms’ i.e. function that perform the required task – “NormalFromMap” is an atom that fetches the normal vector from a map, “NormalFromInput” fetches it as an interpolated value from the vertices. If the material whose shader we want to create has no normal map we simply tell the translator to use “NormalFromInput” for the polymorphic “GetWorldNormal“.

All these atoms are defined elsewhere and could form an entire library. They look like this:

There are many new keywords here.The all-caps words are called ‘semantics‘, they are declared in an appropriate file and indicate the type of the placeholder name and a HLSL semantic name used in case they should be interpolated between shading stages or come as input in the vertex shader. Semantics are essentially variables that the shader translation system knows of.

A sample semantic file looks like this:

Of course if we just substitute parts of the code with snippets we’d be in trouble as different atoms require different data to work with. If we use the “NormalFromMap” atom we would need a normal map, a sampler and uv coordinates. If we use “NormalFromInput” we just need a normal vector as shader input. All functions with an input – that is atoms and the vertex/pixel shader main functions, have a ‘needs‘ clause where all semantics needed for the computation are enumerated.

The declaration/definition(they are the same) of a sample atom is as follows:

atom NORMAL_W NormalFromMap(interface context) needs NORMAL_O, UV, TBN, MAP_NORMAL, SAMPLER_POINT

‘atom’ is required to flag the function. Then the return semantic and the name of the atom. ‘interface context’ is required. Atoms are not substituted by function calls but are inlined in the shader code – to avoid name clashes with vanilla code in the shader that is not dependent upon the translation system all computed semantics (variables) are put in a special structure called ‘context‘. In the atom declaration the keyword interface is used for an eventual future use. Strictly speaking currently ‘interface context’ is not needed but makes the atom resemble a real function and reminds that all input comes from the context. After the closing brace there is an optional clause ‘needs’ after which all required semantics are enumerated.

Sometimes the needed semantics are straightforward to procure – for instance if a normal map is required the system should simply declare a texture variable before the shader main code. However some computations are much more convolved – like computing the TBN matrix. Here comes the third type of resources needed in the translation process – ‘combinators‘.

When the translator encounters needed semantics it first checks if they are not already computed before and are not in the context (I remind you that all data is saved in the context). If it’s a new semantic it checks all combinators for one that can calculate it. Combinators as atoms are functions – their declarations are almost the same as the ones of atoms:

The only difference is the keyword ‘combinator’ instead of ‘atom’. They encapsulate code to compute a complicated semantic from more simple ones.
If no combinatoris found for a needed semantic it is assumed that it comes as an interpolant or vertex shader input. Needed semantic searches are always conducted so combinators can depend on other combinators.

To recap, the building blocks of the shader translation process are:

• semantics
• atoms
• combinators

While it might seem complicated at first, the system simplifies the shader authoring a lot. The shaders themselves become much more readable with no branches in their logic per-material type – so no #ifdef. An atom and combinatorlibrary is trivial to build after writing some shaders – later on operations get reused. The translation process guarantees that only needed data is computed, interpolated or required as vertex input. The ‘context’ structure used to hold the data incurs no performance penalty as it is easily handled by the HLSL compiler. For convenience expanded atoms and combinators are flagged with comments in the outputted HLSL code and enclosed in scopes to avoid name clashes between local variables.

In the next post I’ll explain some compile-time conditions supported by the translator as well as how the translation process works.

 

Follow Stoyan on Twitter: @stoyannk