Deformation and Controlling/Displaying Triangulation in Maya
Maya kinda sucks when it comes to working with triangles. Many a Character Artist arguing (in vain) for your studio/project to switch to 3DSMax will tell you all about it when they’re not complaining about bright lights messing with their Cintiqs and how much it sucks to create hair. They have a point, but it doesn’t have as much of a tendency to mess with us over in rigging world. It did mess with me a bit recently, and being an exclusively self-serving person (service role, schmervice role) I tried to find a solution only when it affected me.
For anyone unaware of what I’m talking about: a quick refresher/crash course. You can be working with a quad model, but under the hood, Maya (and probably about any other 3D anything) will turn those quads (or the N-gons of our sloppier character artists) in to triangles. So why not just triangulate and call it a day? Because triangles suck to work with. You can’t select edge loops, it screws with smoothing algorithms, and just sort of makes things visually crowded as well, making it hard to get those sexy smooth edge-flows you see in all the polycount threads. So most of the time, we either just roll with a quad topology and hope for the best (rarely running in to any incidents) then maybe for those vital things, transfer weights over to a triangulated mesh later. But that approach gets more and more messy as you start to deal with meshes with more than just a simple skinCluster, and get more and more history from other deformers, mesh rivets, or multiple versions of a character rig created for performance speed, etc.
The way Maya and many game engines do it by default is to draw the dividing line between the two closest verts in the quad. Where things start to get complicated, however, is as things start to deform. In Maya, the mesh dynamically retriangulates. You can show the triangles by checking Display Triangles in the shape node. So if two verts were close to each other in bind pose, but through skinning, blendshapes and other wackiness stray away from each other, suddenly your triangle will flip direction. When you see a bit of a visible pop in parts the surface while modeling or deforming things, that’s what has just happened.
That might be all well and good as you want things to always strive to be their best, problem is, game engines don’t tend to do that, so what can end up happening is that things can look fine in Maya, but then out you go to the game engine, your once smooth silhouettes look like bread knives
I ran in to this a bit ago when working on improving the neck deformations for a character at work.
Since for NDA reasons I can’t show this on the actual character (I enjoy being employed) I’ll just show it on an old model I did a while back and dug up and skinned in like 5 unintelligently spent minutes (So don’t judge me). I was working on getting a nice sternomastoid flex on said character when they turned their neck, but the topology wasn’t on my side. Luckily my model I’ll use for this post has the same problem as the model I got at work. But needless to say, at the point where I got the model, it was kind of too late to retopo on a whim for a small feature when the topology is already standard across all Tier 1 heads, otherwise working well, and rigged on multiple heads with hundreds of blendshapes already created per character.
So imagine for the purposes of this exercise that the model actually supports the idea of the top end of the sternum…
So the mesh is dense enough so that you might be able to conceivably make a ridge across the diagonal faces. You could even get things looking good in Maya. But as you might be able to see if you squint, or enlarge the picture, the triangles are doing us no favors. That’s kind of what we’re stuck with in the eyes of the game engine since it locks edges. Thought it would be that easy?
So as we move along, things might look fine in Maya, but remember as you’re making your corrective blendshapes, it’s dynamically re-triangulating.
So say we create a (very) quick corrective blendshape…
So, first problem to solve: How do we see the faces in Maya the same way they’ll be seen in-engine? Well, if you’re saying “Let’s just run triangulate on our main mesh,” you’re wrong….
…. but you’re close.
Problem is, it will still dynamically retriangulate like it did before, because the polyTriangulate is at the end of the input stack, so it’s still not representative. It’s basically doing what Maya was already doing, but giving the triangles solid lines instead of our old implied dotted lines. But as I said, we’re close.
Instead of running triangulate on head_geo, we can run polyTriangulate on the orig shape.
Whenever you apply a deformer to a mesh, it duplicates and hides a version of the mesh that feeds in to the deformer as a baseline connection. So if you select that mesh and triangulate it, then you’ll get that ugly jagginess we were all vying for.
Once you’re done with your nice preview, you can simply delete that polyTriangulate node and go back to working in Quads. Just make sure you delete it before you export. I actually just wrote a tool at work to toggle that I use in these situations.
But frankly, just knowing what it’ll look like isn’t always quite enough. Doesn’t matter how much pushing and pulling of verts you do in pursuit of getting a slim line across perpendicular edges, it’ll always have an angle that has jaggies. Believe me, I’ve tried to find the magic shape that works. So how do we get the control back after the fact and triumph over Maya’s triangular tyranny? It’s actually also fairly simple.
Simply duplicate the mesh in its base pose (you’ll actually wanna make two duplicates. Keep one untouched, and do your work the other), then run triangulate on the faces you want, or do all if you’re a glutton for punishment, and then flip or spin edges as you want (don’t use split polygon. Even if you don’t make new verts, it’ll renumber your verts and break the process) then clear history once you’re done.
Then you take the outMesh of that shape, and connect it to the inMesh of the Orig shape of the head, then delete that connection after it’s taken effect.
And thanks to the second duplicated version we did earlier (hopefully you remembered to do that) you can always easily jump back to the quad version for when you want to do touch-ups on the corrective sculpts or anything else.
Anyways, hope that helps for anyone in a similar jam. I’ve tested it in Unreal and it works there as well with no hitches. It of course also works in Luna (our proprietary engine) but that most likely doesn’t apply to you, unless you work at Insomniac, in which case, hi Edwin (there’s nobody at work named Edwin, but wouldn’t be bananas if we hired an Edwin after I wrote this?)
Until the next random problem comes up that I haven’t seen a more skilled and articulate person solve and write about, causing me to write a rambling bit of nonsense…