Not quite a tutorial
For the stuff I'm making for the upcoming Crumbling Crypt tileset, I'm using a combination of tools, including ShaderMap4. If you want to try it out, you can get a copy at https://shadermap.com/downloads/
For best results, you want to start with a texture without shadowing, but with contours. However, ShaderMap can certainly work with shadowed images. What it can't do is work very well with waxy images, or images with multiple strong light sources.
In general, you can start with any day-lit image though. Let's use this one and see what happens.
The first thing you want to do is clip out the portion you want. Before you do much more, you also want to make sure the image is in the maximum scale you will be using (in your final DDS) and at a size that is fully compatible with NWN. Compatible images are power of 2 (as in 2^n) values such as 16, 32, 64, 128, 256, 512, 1024, and reasonably up to 4096. Each side of an image should be one of those numbers, but it does not have to be the same number. For Crumbling Crypt, I'm just doing 512x512 and 512x1024 images.
Let's turn the above image into a basic 512x512 texture by clipping out the center portion.
Making the texture seamless
If you're not just using the center of the texture, or working with UV island maps (for character or placeable units), you will probably want a seamless texture. There are many ways to do that, but I choose to use GIMP. That program comes with a very basic seamless texture function located under Filters > Map > Tile Seamless...
I'm fairly lazy, so I don't change any of the settings on the popup. Just let it do its thing and give me something to work with.
But you can see there are a few repeating parts in the result. I don't like those, so what I do is put this new image over a copy of my original image (layer 1) as a new layer (layer 2), and clean up the corner regions using eraser.
Erasing the center will usually show you the centers of both images are almost exactly the same. What is important to clean up is the near-corner regions, but don't erase the edge or you lose the tiled-appearance across the border.
My resulting from-GIMP image looks like this (layer 2).
Which results in a texture like this when used over the original (layer 1).
The image is now seamless, and has only minor artifacts of smoothness I don't like. When you do this with a more intentional texture goal, you can be more specific with your erase tool, choosing to keep certain horizontal or vertical lines. But the end goal is to make sure your texture boundaries are shared top to bottom, and right to left.
Other non-standard filters exist for GIMP which allow you to choose only 2-way seamlessness, or to create a floor texture which has four interchangeable sides. But I won't go into those here.
Putting ShaderMap to Work
The first step is to load up the diffuse texture.
Your image will load up and be wrapped to a cylinder. SM4 will automatically make a height map, normal map, ambient occlusion (AO) map, and specular map.
For NWN purposes, they'll all be wrong at least a little bit.
Paid customers of SM4 can add the ability to make a roughness map. On the left panel, click the in any of the empty boxes. Roughness is near the bottom of the list of options.
The first big issue with SM4 output is that the normal map green channel is upside down.
Click the normal map image to switch to normal map options.
Next, click the axis icon in the bottom left panel.
Next, click the box at the top of the green axis.
Click the checkbox to accept the changes.
Post Processing Work
When you're done with the above steps, you have produced some very basic maps for use in NWN. However, they're all somewhat wrong.
Next you will want to import them into your favorite image editing program. I use Paint Shop Pro (PSP7).
The following discussion will be based on workflow in PSP7, but many blending filters can be found in other program which will be nearly the same.
The first step is to export all the maps from SM4 to PSP7. Each map box has a button specifically for this purpose.
What it does is opens a PNG copy of what you see in SM4 into your art program.
Any changes you make to the PNG file from your art program will also be seen by SM4 and loaded into the viewer.
The next step is to modify each layer individually so it looks better in NWN.
Let's do the normal map first.
A little bit of normal map goes a long way to making things look dark and shiny. Exports from SM4 are generally not deep enough, so we need about 50% more.
Duplicate the exported image as a new layer, and set it's opacity to 50%. Next set the blend mode to "Overlay".
In PSP, "OVERLAY" is the only way to retain correct HUE when you blend normal maps. Otherwise some of the pixels in your final image will be pointing the opposite direction and will pick up weird glares from lighting.
(Pstemarie) I've found that for stone, the settings used for the normal map (Overlay 50%) didn't quite get the depth I wanted. Instead, for smooth, unpolished stone use Overlay 100%. For courser stone, duplicate the Overlay layer again, using 100% opacity. Continue duplicating until the stone is of the desired depth.
Next, let's mix the AO map into the diffuse map.
Make sure to have a backup of your original diffuse map in case you make a mistake or want to make an alternate texture.
Copy the AO map layer from SM4 as a new layer over your original diffuse map.
Keep the opacity at 100%, and set the blend mode to "Multiply".
This bakes some of the wanted AO into the diffuse map. This step represents some of the self-shadowing.
Depending on your original texture, the AO map will almost be pure white, so your output in this step may look exactly the same.
This step is generally cosmetic. It's not a requirement, but I think it helps a lot on plain textures.
Next, let's do the height map (displacement map).
In NWN, height map pure white will be equal with the surface of the mesh. Anything lower than white will be inset.
This is contrary to the name and somewhat unintuitive.
The best thing to do would be to find a math approach and then subtract the negative values of your height map from white. However, PSP won't do that easily.
Instead I'll use multiple layers to achieve a similar output.
First duplicate the SM4 height map as a new layer.
Set the blend mode to "Screen".
Duplicate that layer again as a third layer.
Next, add a new blank layer and paint that layer pure white. Set the opacity of that layer to 50%.
Your output should look something like this and will make a better height map without a glass lens effect at the edges.
If you want deeper pits and cracks, simply copy the merged result and add it as a new layer with blending set to "Multiply". Adjust or duplicate that new layer as needed.
Next, let's do specular and roughness together, since they're similar.
On the specular map, duplicate the layer and set its blend mode to "Multiply'.
Copy the new layer again. You can adjust this second copy's opacity to get the desired darkness you want.
On the roughness map, duplicate the layer and set its blend mode to "Screen".
Copy the new layer again. You can adjust the second copy opacity to get the desired brightness you want.
(Pstemarie) For the roughness maps, I found that it was easier to control the gray shading by following the procedure above but using 100% for the opacity of the 2nd copied screen layer. Finally, add a "brighness/contrast adjustment layer", setting the value of the brightness to 50%.
A general rule is to set the second layer opacity about the same on both.
However, for a flatter texture, make a brighter roughness map. And for a wetter texture, make a darker roughness map.
On a higher roughness map, higher specularity will still force more of the color of a torch light into the texture. Very dark specular maps will have very little color added.
Next, you will want to export these images to your override or development folder and make a material file.
Since GMAX has crap for texture memory, and will often crash if too many textures are in a scene, I also make a much smaller PNG file of my diffuse map.
1/16 the size of the original is a good start (32x32 for a 512x512 texture). If you are using Blender or newer MAX, this should not be an issue.
Your material file should look like this for a texture kit named "tutorial".
Almost every texture in my cave uses this workflow.
I find it handy to have multiple torches in my inventory with a testing character. I keep multiple colors and brightnesses handy in my quickbar. I make a set of item blueprints for each torch and carry them to each module I use for testing. That way the torches and light rings all have an item name including the color and brightness right on them. Already identified and ready to use.
Assuming you have this texture applied to something you can see in a module, load up the module in NWN and go to painted object.
On the resource manager panel, there is an option to reload textures. It has a danger warning, and honestly I have had this crash a few times out of thousands of attempts, so it's rare.
You can now make changes to your textures in your art program and then near-instantly load the changes in the running game to test what you did.
I suggest making small changes to each map to see what it does and by how much.
Finally, when you get the shape and depth of your maps OTHER than diffuse worked out the way you want them, go back and modify your diffuse map LAST.
Things I do to my diffuse map include brightening, some shadow removal, addition of color noise, etc.
You can brighten your diffuse map easily by copying the original as a new layer, colorizing it to black and white, and then inverting it. Use various blending filters such as screen, overlay, dodge, or hard light.
You can reduce saturation on your diffuse map easily by copying the original as a new layer, colorizing it black and white, and then setting the blend filter to color or hue. Change the opacity to carefully remove color.
You can remove some depth and shadow by copying the diffuse map, colorizing to black and white, inverting, and setting blend mode to exclude. Change the opacity to carefully remove depth and shadowing. If needed, you can also add another copy of your AO map to add back specific good AO to your de-shadowed image.
Side Note: Splitting a finished texture away from the original background for use as a decal
Let's say you have a custom texture you made from painting with one texture over another, like below. (Images from Tonden Ockay)
But let's say you want to use the same paint job over another background. You can use this technique to lift the paint job off as a decal which you can then apply to other backgrounds, or use as an in-game decal using transparency.
Again, my example will be done in PSP7, but I assume all major art programs have these functions.
The first step is to put the original background into a new image as a layer. Next, put your final image in as a second layer. Set the second layer blend mode to "difference".
|Layer 1||Layer 2||Result|
|Blend Normal||Blend Difference|
Next, copy the result into a new image. Now create a mask using the image as a source. Choose the "any non-zero value" option.
Your result should look something like this.
Now save that mask to your alpha channel (found on the same menu as above). Then delete the mask (same menu) with the option to NOT merge to the layer.
Next bring in a copy of your final texture as a top layer. Your image can now be saved as a reusable decal texture.
To test your decal, simply load your alpha channel back in as a mask to the final texture, like this. Just make sure not to save it like that or the black color will mess up the decal.
As options, you can also reuse just the alpha channel with a different brick or rock texture over a different wall texture.
You can also use different options to make your mask, such as source luminance, which will produce a fainter but softer alpha channel.