Creating Topiary and Realistic Plants

Advanced Tutorial

01. Introduction – What you’re going to learn

This section will be a tutorial on how to build plants – whether fake or real. For both approaches, we will model a single branch with leaves, then pattern that branch around a shape – whether that shape is an actual tree or a sphere. This tutorial will walk through the process for creating a realistic plant first, then it will go through the basics of creating a plastic one. The example plant we will be imitating will be a round-leafed eucalyptus.

02. Realistic & Fabric Plants: References

To start with, we’re going to collect photo references of the plant to be modeled. These can be found online or if you have the plant handy, you should just snip off a leaf and scan the front and back of it in an office scanner for best results.
Note that this technique will also work with fake plants that have fabric leaves.

a. Photo References

I was able to source these reference images – these need to be as high resolution and as much of a flat view as possible. We will be using them as textures for our leaves and it is critical that they are sharp and crisp – otherwise nothing we do will make the plant look real. It is also important that they have as little perspective distortion as possible. The first image on the left is a good example, the leaves on the far right are an example of what not to do.

Image Credits: SFA Dendro


I will be replicating this arrangement that I found online. Note that it’s not critical to get the exact placement of branches – it is expected that an organic product like this will have some variation between the source photos and the received product.

03. Realistic & Fabric Plants: Preparing Images

Now we’ll take these leaves and turn them into a usable set of textures. We will do this by picking a leaf with minimal distortion and the highest resolution we can find.

a. Isolate the Leaf

Using the “Quick Mask” mode (shortcut toggle: Q) and/or any of the selection tools, select the single leaf that you would like to use. If you have never used Quick Mask before, it is a separate mode that allows you to paint your selection using the Brush tool. Here is a short tutorial (Credit: How to Gurus, Youtube). Note that red means the area that you will not be selecting once you turn off Quick Mask. To select the inverse, select the Marquee tool (Shortcut: M), right click, and select “Select Inverse”. Make sure to refine your mask as much as possible, having a tight selection is very important.

When selecting your leaf, make sure to include at least some of the stem that connects it to the branch.

b. Crop & Remove Background

Right click on your selected leaf and select “Layer via Cut”. This will place the leaf on a separate layer. Delete the background layer to end up with just the leaf. Note that you may have to disable the Lock icon on the layer in order to delete it. To disable it just click on the icon as it appears next to the layer.

Afterwards, add a bottom layer that is completely black and crop your image as close to the edges of the leaf as possible.

c. Remove Lighting Information and Ensure Color Accuracy

If you have any lighting information from a camera flash or scanner light then it needs to be removed. Additionally, shining a bright light on a leaf can often change its color, sometimes drastically in the case of this example. You will need to color correct the leaf to make sure it looks like it would in normal lighting conditions.

Note: We will make many textures in this one source file. It is recommended that you put all of your changes in a single folder and name it “RGB” or “Diffuse”. Leave the black Background layer outside of this folder.

d. Create the Opacity Map

You can use an Opacity map to tell V-Ray where to show a material and where to hide it. This is different from a Translucency Map. You can think of the difference like so: a Translucency Map shows where a material is see-through. An Opacity map shows where the material has been cut away, like a stencil. White is where there will be material, black is where it will be completely invisible.

Copy your RGB folder. You can do this by clicking and dragging it onto the “Copy” Icon at the bottom of the layer stack. Then delete everything except the base image layer. In the image to the right you can see that I have the RGB folder hidden and just the LeafImage in the Opacity layer.

Double click on the LeafImage layer in the Opacity folder to open the Layer Effects dialogue. Check “Color Overlay” and select white as the color. You should end up with a black and white image of the leaf:

Sometimes you might get a “halo” around your image when it’s applied to the model. This is because the opacity map might “bleed” a little, depending on how closely cropped it is. To fix this you can use the “Stroke” effect to add or reduce the area of the mask. Enable the Stroke effect. To shrink the border of the mask, select Position: Inside, Blend Mode: Normal, and 100% Opacity. Then set the color to black. By increasing the size, you can shrink the mask to hide any sort of halo-effects you may have.

e. Make the Bump Map

We will need a bump map to provide microsurface details to the render engine. Since we’ve already flattened the lighting information in the diffuse map, we can just take all that work and generate a bump map from it.

Duplicate the RGB folder and merge all of the layers inside of it (don’t forget to rename the folder for the sake of your sanity later!). A quick shortcut to doing this is selecting them all using shift-select and pressing Ctrl-E. Make sure that your image does not use the background layer – it should be transparent around the leaf.

Then, select the new layer and use the Bump Map filter under Filter -> 3D -> Generate Bump Map. For a flat leaf image it is recommended that you switch the preview object to a Cube. Adjust the settings as you wish to create the bump map, then hit OK.

f. Make the Translucency Map

Most real and fabric leaves are semi-translucent. It’s what gives them their leafy look. To create this effect we will be using a V-Ray 2 Sided Material, which uses a texture to determine which areas are transparent (white) and which are not (black).

With a leaf we want the veins and stems to generally not be translucent, whereas the rest of the leaf is slightly see through. To make the translucency map, start by duplicating your RGB folder and renaming it to Translucency or Transparency. Merge all the layers like you did to create the Bump map. Then make the layer black and white. You can do this by selecting the layer and pressing Ctrl+Shift+U on the keyboard.

Next, you’ll want to make sure that your stems are black and your leaf is white. In the case of this image, I had to invert it (Ctrl+I). Now the stems are dark and the leaf itself is light. However the leaf is extremely white – it will be very transparent. We want more of a light grey.

Add a curves layer to make sure the leaf isn’t too white.

The image ended up looking a little too washed out. So let’s make a duplicate of the curves layer and the base transparency image, then merge them together (select both layers and press Ctrl+E).

Set this new layer’s blend mode to Color Dodge. This will bump the contrast overall. Now just play with the opacity to make sure there are no areas that are pure white. On the left we have the Opacity at 100%, on the right it’s at 50% to create the final image.

All final images can be downloaded here.

04. Realistic & Fabric Plants: Modeling and Basic Texturing

Now that we’ve got some textures we’re going to jump into Max and make our leaves.

a.Create a Plane

Under the Standard Primitives tab, create a simple plane and center it in the world. For its length segments and width segments settings, set it to 4 and 4. For the length and width, size it to be the same aspect ratio as your leaf image. To do this you can just type in the pixel dimensions of your image (for example: 1640 x 2193) as the length and width of the plane. Then just scale it down to a more appropriate size (for example, eucalyptus leaves are usually about 2-4” long). While it’s not critical to have an exact size, you can check the dimensions of your model by selecting it and going to the Utilities panel (the hammer icon) and picking “Measure”.

b. Apply a Planar UV Map and Create a Basic VRayMtl

Apply a planar map. Make sure it’s the same size as your plane. You will have to uncheck “Real-World Map Size” in order to edit the UVW Gizmo.

After creating the planar map, create a basic V-Ray Material, plug in the textures from before and assign it to the leaf model. Remember that the Transparency map is only going to be used for the 2 Sided Material, so you don’t need it yet. You may need to enable seeing texture maps in the viewport. See the image to the right for how to do this.

Once you have created the basic material, set up the specular and bump values to match what the leaf looks like in real life. Generally the top of a leaf is more reflective and has tighter specular highlights than the back. For this leaf I used a value of .8 and a medium-dark grey of 50. Finally, I added a grunge map into the Refl. gloss channel, to give the illusion that the leaf is dirty.
Note: Ignore the fact that my images are showing as all black – it’s a 3DS Max glitch. They’re there.

You might get some artifacts if your UVW Map is slightly smaller than the plane you’ve created. Your images will start to tile and this can produce some strange effects with slightly cut off leaves. To fix this, just double click your Diffuse and Opacity textures and uncheck tiling in both directions.

c. Shape the Plane

Next, give the plane some shape by giving it some depth. Drop the center down to create an arch and bend the entire plane. This will just make the leaf look less flat in the final image. While you’re doing this you can check off the “Show end results on/off toggle” on the Modifier List so you can get an idea of how your changes are affecting the image mapping.

05. Realistic & Fabric Plants: Creating a 2 Sided Material

Now that we’ve set up the basic material and model we need to set up the 2 Sided Material.

a. Lights, Camera, Action!

You will need to either use our pre-made render scene or set up your own. The most important thing is to have enough light coming from near the camera so that you can see the front of the leaf, and a light behind the leaf so that we can evaluate the 2 sided Material. The setup I am using is below – on the left is the camera view, in the middle is the render scene, and on the right is the overall scene setup. My lights are all set at a multiplier of 200 and my camera’s shutter speed and ISO are both at 100. Using the Interactive Preview Render (or IPR) will really help you with these steps.

b. Duplicating the model

To truly test our leaf, we have to see both its top, bottom, and side. To do this easily, create two instances of the leaf. An instance is an exact duplicate of an object where any changes made to the original object will also change the duplicates. Once you’ve made the duplicates, rotate them so that one has the bottom facing the camera and the other has both the top and bottom slightly visible. Bonus points if you can get the leaves to be very close to each other and overlap slightly: this will help a lot when checking the translucency next.

c. Setting up the 2 Sided Material.

Create a Vray2SidedMaterial in the Material browser. Plug the basic material that we just made into the frontMtl slot. Then add the Transparency texture we created into the translucencyText slot.

Place an Output node between the Transparency texture and the 2 Sided Material node. This will allow us to adjust the intensity of the transparency (essentially a curves layer inside Max). To adjust the node, double click it and check “Enable Color Map”. Then right click on one of the dots on the curve graph and select Bezier-Corner. Now you can drag the handle to adjust the intensity of the map. Below, I’ve boosted the whites.

Note: Ignore the fact that my images are showing as all black – it’s a 3DS Max glitch. They’re there.


Next, duplicate the basic material we created before and plug it into the backMtl slot. This material will be what shows up in the white areas of the transparency map.

Adjust the backMtl to have a wider gloss value (I used .5) and slightly less reflection (I used an even darker grey of 24).

Finally, go into the VRay2SidedMtl and make sure to check on “Back material:”. This will tell the 2 Sided Material to use the VRayMtl we plugged into the backMtl slot. Do a test render to see how it comes out!

Now it may be difficult to tell exactly how transparent your leaf is and where it’s transparent. To test, remove the Diffuse map connection to the back material and set that material’s diffuse to a very bright color – red is a good one to use. Now you’ll see where the back material shines through. In the image below, the leaf furthest away from the camera is showing us its bottom side – so it is completely red. The leaf in the middle is showing us its top and we can see red in some areas and not others. The leaf closest to us is on its side.

Once you’re satisfied with the transparency of your model (which can be adjusted via the Output node on the transparency map), re-attach the back material’s diffuse.

One other thing that we can do is slightly color the back of the leaf by dropping the Diffuse map value in that material to about 80%. Then, change the Diffuse color of that material to a slightly lighter green. This will blend with the diffuse texture we’re using and allow for a different color to the back of the leaf. Once you’re done making the leaf material, delete the other leaves and move the original one off to the side for later use.

06. Modeling the Stems With Leaves

If we take a look at our source image, we’ll see that for this tree (and many others) leaves sprout off of long stems. What we will do is model these stems with their leaves, and then pattern them to create the full foliage.

Note that the following technique doesn’t just work for linear stems. It can also be applied to make small branches with leaves on them, and then patterning those around the object to form the final result.

a. Create a Cylinder

This is pretty easy to do. In the same scene as before, create a tall, thin cylinder primitive. Give it enough height segments to add some bends without looking too jagged.

b. Add a Few Modifiers

The Taper modifier will narrow the cylinder for you so you don’t have to do it by hand. In the example to the right I have tapered the cylinder by -0.5.

Next, add a UVW Map modifier and set it to Cylindrical.

Then, finally, add an Edit Poly modifier to the stack. Right click on the modifier and select “Collapse To”. This will turn the model into an Editable Poly and absorb all of the changes we just made. Say “Yes” to the dialogue that appears.

c. Shape the Stem by Hand

Now grab each of the loops of verts and move them around to give the stem a little bit of waviness. Leave the bottom of the stem at 0,0,0 world space however. We will be creating a pivot at the base later and it’s better if this doesn’t move. You should delete the bottom face of the stem as well for when we smooth the model later.

d. Assign a Basic Material

Stems have a pretty simple material. In this case it’s just a flat green with a small amount of reflection and a high RGloss value. Adding the material now will make the next step a little easier.

e. Add the Leaf to the Stem Object

Now let’s add the leaves to the stem. We’re going to put them into the same Object because it will make 3DS Max perform much better when we copy this object hundreds of times. Select the Stem, then find “Attach” in the Edit Poly rollout. Activate the tool, then click on the Leaf. The Attach Options dialogue will come up. Select “Match Material IDs to Material”. This will change the Material ID of the Leaf to a different number and automatically assign a Multi-Sub Object Material to the new multi-element object. This is why we added a material to the Stem before combining it with the Leaf.

Go into the Material Editor and use the Pick tool to sample the now-combined object. You will find the newly created Multi-Sub Object Material automatically created and assigned.

f. Duplicate and Arrange the Leaves

We can now start to arrange the leaves along the stem as duplicates. Select the leaf as an element and while holding Shift, click and drag on Z to duplicate it several times along the stem. Make sure to select “Clone to Element”.

Since the leaves have a VRay2SidedMtl applied to them, their texture maps won’t show up in the viewport. This can make it difficult to make sure that the stem of each leaf is actually on the main branch. To make it easier, temporarily unplug the VRay2SidedMtl from the Multi-Sub Object Material and plug in the top material instead. Just remember to plug the 2 Sided Material back in at render time!

Duplicate the leaves several times along the stem, in opposite pairs. Scale them up as you go. For this tutorial we’ll add five sets of leaves, to mimic the spacing of the leaves as shown in the reference. Additionally, each leaf should be rotated along its axis, to maximize realism and minimize angles where the leaf would be seen edge-on.

g. Add Variation with Duplicates

To increase the realism when we get to patterning this stem, we should also pattern variations of it – that way the repetition will be less visible. So create copies (not instances) of the stem and adjust them slightly to get a couple variations.

Once you’ve added all the leaves, don’t forget to plug the VRay2SidedMtl back in! Move the stems off to the side for now, we’ll need them again in a few steps.

07. Modeling the Tree

Now we need to make the bulk of the tree itself. This will be the object that we distribute the stems onto. This tree in particular has a long curving trunk with a few smaller branches coming off of it. It also has several stub branches that have been trimmed near the bottom.

a. Create a Curve that Matches the Trunk

Under the “Shapes” tab on the Create panel, select the “Line” tool. Make sure the Initial Type and Drag Type are both set to Smooth. Then draw the trunk of the tree. You will have to draw the trunk laying flat on the grid, and then rotate it 90° to stand up straight. Hint: Press “A” to enable Angle Snapping while rotating. Don’t worry if your trunk isn’t the right height, you can always scale it later.

b. Create a Cylinder

Create a tall cylinder that is about the height of the line you just drew, and has a width similar to the trunk. Give it a good amount of Height Segments as well. Add a Taper Modifier to the Cylinder. The value of the Taper on this cylinder is -0.5. Don’t forget to add a UVW modifier as well!

c. Add a Path Deform

Add a PathDeform(WSM) modifier to the cylinder. Then click the “Pick Path” button, then click on the Curve you drew. Your cylinder might end up in the wrong spot or looking strange. If that’s the case then press the “Move to Path” button. You can use the “Stretch” value to stretch the geometry to fit the length of the path if it’s a little short or long.

d. Start Making Branches

Use the Line tool to draw a couple branches made of separate lines. Make sure to also rotate the branches in 3D as well (don’t just draw them flat on the ground plane). Try not to do too many of these, you only need two or three unique branches.

Then create a cylinder like before – make sure to add the modifiers but don’t pick a path yet.

e. Copy the Cylinder and Pick Paths

Shift+Click and Drag the cylinder to create a clone. When the Clone Options dialogue comes up, select “Copy” and set the number of copies to be equal to the number of lines you drew.

f. Attach the Cylinders to the Paths

Select each cylinder and go to the PathDeform(WSM) modifier. Pick the path you want to attach it to, then select Move to Path. Use the Stretch option to scale the cylinder to fit the chosen path and use the Taper and root Cylinder modifiers to adjust the taper and radius, respectively.

g. Convert Cylinders to Edit Poly

Select all the cylinders and Right Click -> Convert to Edit Poly. Then go through each cylinder individually, right click on the PathDeform modifier and select “Collapse To”. Say “Yes” to the dialogue that appears.

h. Attach all Cylinders to a Create a Single Object

Select one of the cylinders and click the “Attach” button in the Edit Poly modifier. Select all the smaller branches to create a single larger branch. Note: If you want to physically join these meshes together it will be easier if you use cubes instead of cylinders. But it’s not necessary unless you can really see the branches.

i. Move the Pivot of the Branches to the Base

Select the now unified branch and go to the Hierarchy tab. Select “Affect Pivot Only”, then move the pivot to the base of the branch. Once you’ve done this, click “Affect Pivot Only” to deactivate the mode.

j. Enable Snapping

To put our branches on the main trunk of the tree, we’ll need to enable 3D snapping and snap-to-faces. At the top of 3DS Max find the Snaps Toggle button. Click and hold on it to make sure it has a little 3 next to the icon.

Then right click on the icon to open the Grid and Snap Settings dialogue. Select “Faces” and “Vertices”, then close the dialogue (X in the top right).

k. Position the Branches on the Trunk

By using snapping we can make sure that our branches stick to the trunk. You may need to enable “Use Pivot Point center” in order to use the pivot locations of the branches.

l. Attach Branches to Trunk as Elements

Once you’ve completed positioning the branches around the Trunk, convert the trunk to an Editable Poly using the same technique as in Step G. Then attach all the branches to the trunk using as elements. Before you do this it’s a good idea to copy the branches and move them aside, just in case you need them later.

To add the small bumps where branches have been cut off, create a few short, tapered cylinders and place them around the trunk in a similar manner. Convert them to Edit Polies and Attach them to the trunk the same way we did the branches. Make sure to move the branches and bumps so they intersect with the trunk rather than sitting just on top.

08. Creating the Bark Material

Generally if you can find a texture that is the right kind of tree bark for your trunk then you should definitely use it to generate your Diffuse, Reflect, Bump, etc. maps.

In the case of this tree, I could not find an image of the exact same kind of bark – so instead I focused on finding something that had a mostly smooth texture to it.

a. Find an Appropriate Texture

I found this smooth bark on (middle image, right), which looks close enough to what this tree might look like in real life.

I then modified it using Photoshop to match the general color of the eucalyptus tree (bottom image, right). If you want to check out how I did this, you can download the Photoshop .PSD here.

b. Create a Basic Material & Apply it in 3DS Max

Apply a basic VRayMtl to the object. Using the Interactive Preview Render (or IPR) will really help you with these steps.

c. Building the Diffuse

With this particular tree the color is generally pretty green, but fades to a very dark, almost black as it goes around the edges. To mimic this effect, we’ll use a falloff node from green to dark green/black. To add in a little extra texture detail, we’ll use the texture from before mixed with the green, blurring it a little in the bitmap node to make sure we don’t see any sharp details.

To make the tree easier to see you can change the background color of the render. Press 8 on the keyboard and change the Background Color to white.

d. Adjust the RGloss Values and Create a Bump Map

Trees with rougher bark will need to have a tiling bumpmap and well UV’d branches. In this case we can just get away with a BerconNoise since the tree is so smooth. The tree is also relatively reflective with a tighter highlight, so the material is adjusted as well. The Reflect value is set at a 28 grey with an RGlossiness value of 0.5 and a bumpmap value of 10%. The BerconNoise is default except the size is 0.1.

That’s it! We can always add more detail later but most of the tree should be covered by the leaves, which we will add to it in the next section.

09. Preparing the Leaves & Stems for Duplication & Distribution

This section will cover distributing the stem/leaf models across the tree. Before we do this however, we will need to prepare them for a physics simulation, which will make sure that they behave realistically and do not intersect each other.

a. Enable the MassFX Toolbar

The MassFX Toolbar is where 3DS Max’s physics tools live. To enable it, right-click on an empty area of the menu and find it. Once you click it it will appear as a floating window, which you can dock on your toolbar for ease-of-use by clicking and dragging the top of the window onto an empty area of your toolbar.

b. Adjust the Pivot of the Stems

Place the pivot of the stems at the base of the stalk. Do this for all copies of the stem.

c. Apply a Dynamic Rigid Body Modifier

A Dynamic Rigid Body in Max will behave like any normal object in the real world – it is affected by gravity, other objects, forces, etc. This is in contrast to a Kinematic Rigid Body, which are animated and can affect other objects but not be affected by them and Static Rigid Bodies, which can’t move but can affect other objects. For more details, check out 3ds Max’s help documentation (Autodesk).

Since we’re going to do is “jiggle” the leaves into place so that they don’t intersect with each other, we will need them to affect other objects and be affected themselves. Click on a stem and add a Dynamic Rigid Body modifier by clicking on the icon in the MassFX toolbar. This will add a MassFX Rigid Body modifier to the stack of that object and the object will have a transparent white mesh around it (the collision mesh).

d. Refine the Collision Mesh

3ds Max will automatically add a collision mesh to the object. This is a simplified version of the main mesh that is used for quicker calculations. However the default version is a bit too coarse for leaves – we will have to generate a new one. In the MassFX modifier, scroll down to the Physical Shapes rollout.

Under “Shape Type”, select “Concave”. Then scroll down to the Physical Mesh Parameters rollout. Press “Generate” to have 3DS Max auto-generate a concave collision mesh. Since the stem models are pretty simple the default settings are fine. Note: on more complex models it’s a good idea to save before generating a mesh, sometimes it can crash.

Once Max is finished the new collision mesh will appear. Repeat this process for any number of stem variations you have.

e. Add a Ball and Socket Constraint

The objects are now ready to be simulated – but they won’t give us the behavior we want. If you press the Start Simulation button now, they will just fall to the floor.

Note that after every time a simulation is run it must be reset before making changes to the base mesh. The reset button is to the left of the play button.

In order to make sure that the stems behave properly and are rooted at their base, we will need to add a ball and socket constraint. First, select a stem. Then select the ball and socket constraint from the constraints drop down on the MassFX toolbar. Note that the location of the constraint will automatically be based on the location of the object’s pivot.

It might appear very large at first. Just scale it down by moving your mouse or using the scale tool. The actual size of it does not matter – it is just a visual representation. Making it smaller just means that it does not block the view. Repeat this step for all copies of the Stem.

f. Group Constraints & Objects

In order to pattern these objects and their constraints, we will need to group them together. Select each stem and its constraint and group them. By the end of this section you should have two to three separate groups, with one stem and one constraint in each group.

g. Reposition the Group’s Pivot

Very important. Once the objects have been grouped they will have a new pivot. Enter the Affect Pivot Only mode and move the group’s pivot to the base of the stem – where it was before.

A fast way of doing this is to use the Align Selection tool. Press Alt-A and your cursor will turn into a crosshair whenever you mouse over another object. Click on the constraint at the bottom of the group. The pivot will automatically align to it. Make sure that you select X/Y/Z Positions and Pivot Point/Pivot Point in the dialogue box that appears once you click.

h. Create a Collision Mesh for the Tree

Select the tree trunk itself and then go to the MassFX toolbar to make it into a Static Rigid Body.

Now, depending on how complex your tree model is, using a generated Concave collision mesh might take a long time to calculate.

So instead, duplicate your tree model and remove the MassFX Rigid Body modifier from the copy. Make sure that if you move the copied model away from the original you do it only on one axis.

Select the copied tree and add a ProOptimizer to reduce the polycount. In this example I used a 30% Vertex level, which brought it down from about 21,500 faces to 6,300 – much better for physics calculations. You will have to press “Calculate” first, then change the Vertex %.

i. Assign the Low-Poly Model as a Collision Mesh

Select the original tree trunk model (the one with the MassFX Rigid Body modifier). Scroll down to the Shape Type dropdown and select “Custom”.

Under “Physical Mesh Parameters” select Pick Mesh From Scene and click on the new low-poly mesh.

This will add the low-poly mesh as a Collision Mesh. Once this is done you can delete the low-poly mesh. However, once you click on the tree trunk object you’ll now find that its collision mesh is off-center.

Go to the MassFX Rigid body modifier, select “Mesh Transform” and use the move tool to move the mesh back into place on top of the main object. It doesn’t have to be exact but the closer the better.

10. Duplicating & Distributing the Leaves and Stems

Now it’s time to duplicate the objects and distribute them onto the tree.

a. Create a Crowd Object

We will use the Crowd Helper for this task. It can be found under the Helpers menu. Create one by selecting the “Crowd” helper and clicking and dragging in the workspace. The actual size doesn’t matter but don’t make it very large – just enough to see. This way it won’t get in the way when selecting other objects. Once created, center it in the world at 0,0,0.

b. Open the Scatter Objects Panel

With the Crowd object selected, go to the Modifier list and select “Scatter”. This will open the Scatter Objects menu. Note that once this menu is open you cannot rotate or change your view in the Viewport – so you’ll need to set up a good view beforehand while you work. Note: this tool will use the original source objects as part of the scatter. If you want to preserve them for later editing, make a copy of them before proceeding. You can always use one of the clones but it might make it easier if you have the originals sitting nearby.

c. Set Number of Clones

Click the button immediately under “Objects to Clone” and select one of the groups that you created before. Then type in how many you want – remember that we’re going to be repeating this process with the other groups, so don’t go overboard. Since I have 3 copies of the stem, I have chosen to make 25 copies of each (for a total of 75 stems). Make sure that the cloning method is set to Copy (instances won’t work with constraints) and Clone Heirarchy and Clone Controllers are both checked.

We’re not going to click “Generate Clones” yet since we can do all the operations all at once later.

d. Set Positions

Next under the “Positions” tab, choose “On Surface” and select the Tree from the list that appears. Then make sure to uncheck “Include childrens’ bounding boxes in spacing calculations”. This option will try to make sure that objects aren’t intersecting but it can cause issues with large amounts of objects. However it seems that it still factors in the number in the box, so change the spacing value to something very small like 0.1.

e. Set Rotation

The default settings on the Rotation tab are usually fine. Add a bit of Sideways and Up/Down deviation just to add some variety to the initial placement.

f. Set Scale

For even more variation we will randomly scale the clones. Under Y and Z, set them to be “Same as X” so that the objects are scaled uniformly. By default Scale is 1.0 and Deviation is 0.0. This means that your stems will be 100% scaled by default and all of them will be the same size. Increasing the deviation will increase the potential range – so for example a deviation of -0.2 means that all the clones will be between 80% and 100% similarly, a deviation of +0.2 means that clones will be between 100% and 120%.

For this example I have a Scale of 1.0 and a Deviation of -0.2. Note that my Y and Z are set to “Same as X”.

g. Clone Objects

Now it’s time to scatter the objects. Under “All Ops” check off Clones/Positions/Rotations/Scales. Then press Scatter to scatter the stems. This will perform all the actions from before at once. Once the clones are generated, press OK to accept the results, or Cancel to delete the clones and close the tool. You can also go back and edit any of the other tabs and regenerate them.

Note: if you encounter an error like the one below, go back to the Positions tab and lower the Spacing value.

h. Refine Object Placement

Some of the stems have patterned onto areas that we don’t want them – like the tree trunk. So re-enable the 3D snapping tool, select the individual stems and move them up the trunk. Make sure they don’t intersect other stems as much as possible.

i. Repeat for Other Copies

Repeat the above steps for the other copies you made of the stems. Note that you’ll have to hit “OK” once you’re finished patterning each stem to accept the clones. If you feel that the leaves are too close together you can always remove some, rescale them, or move them apart from each other.

11. Simulate

Now let’s run the simulation. The main goal of the sim is to make sure that the leaves are behaving properly and don’t intersect with each other.

a. Set the Sim Parameters

To start with, open the MassFX Tools menu from the toolbar. It is the globe icon with the checklist next to it.

Most of the default settings in this menu are fine, we just care about four of them: Gravity, Use Ground Collisions and Solver Iterations/Substeps.

The more gravity there is, the more our leaves will droop, and the less important collisions between two different stems will be. The lower the gravity, the fluffier the tree will become. No Gravity is useful when all you want to do is spread leaves out from each other and keep them from intersecting.

Also, turn off “Use Ground Collisions” since we won’t need them.

Meanwhile, Solver Iterations are the number of times the solver enforces constraints and collisions. The higher this is, the more detailed collisions will be – but generally anything over 30 is overkill.

Substeps are the number of times in between frames collisions, etc. are calculated. From the 3ds Max help documentation (Autodesk): “(Substeps + 1) * Frame Rate. A frame rate of 30fps with 0 substeps results in 30 simulation steps per second, 1 substep results in 60 steps per second, 2 substeps results in 180 simulation steps per second, and so on. “.

For this simulation let’s select No Gravity and set the Substeps to 1, and the Solver Iterations to 10.

b. Set the Timeline

Simulations are run along the length of the timeline. To increase the amount of time a simulation has to come to a solution we have to modify the timeline’s length. There are two ways to do this:

  1. Hold Ctrl+Alt+Right Click and Drag on the timeline.
  2. Use the Time Configuration tool on the bottom right of the screen:

Also accessible by hitting X to open the Search All Actions dialogue and typing in “Time Configuration”.

Within the tool you can type in the number of frames. Let’s change it from the default of 100 to 200 by changing the End Time. Press OK.

c. Run the Simulation

Press the Play Simulation button in the MassFX toolbar to start the simulation. Keep an eye on the timeline (bottom of the screen). Once it reaches the end, press the Play button again to stop the simulation. The simulation will keep going past the end of the timeline but anything that happens after the last frame won’t be saved.

To erase the simulation and reset it, press the “Reset Entities” button which is immediately to the left of the play button. Now you can reposition the objects and run the simulation again.

d. Using Capture Transform

Note: This step is not always necessary. You may want to make changes to the initial position of the stems based on results of the simulation. From the 3ds Max knowledgebase (Autodesk):

For example, you might use this to start a simulation with a stable pile of bricks. You could create a bunch of bricks in the air, run the simulation, wait until everything settles, and then invoke this menu command. Thereafter Reset Simulation would always return the bricks to their new, piled-up locations [before the next simulation].

To change the initial rotation and position of an object, select the objects that you want to capture the simulation data for, then open the MassFX Tools dialogue and go to the Simulation Tools tab.

With the objects selected, press Capture Transform. Now if you press “Reset Simulation” the object will retain its new translation information. This is useful if you want to run the simulation and then re-adjust the positioning of a few leaves. So for example you could Capture Transform all the leaves in a simulation, change the positioning of a few of them, and then capture their transform. When you run the simulation again, everything will be in its new position to start with.

e. Bake the Simulation

Once you’re happy with the results of the sim, it has to be baked into a normal animation in order to save it. In the MassFX Tools menu, open the Simulation Tools tab and select “Bake All”. This will replay the animation, saving all the transforms to keyframes on the timeline in the form of green and red ticks, which you can see by selecting a simulated object:

Once this is done, select all of the animated leaves and only the animated leaves. It will be helpful to use the scene browser to do this. If done correctly all the keyframes for the selected objects will appear on the timeline.

Next, you will need to delete all the keyframes except the last one. This can be done by clicking and dragging on an empty area in the timeline to select the keyframes and then hitting the Delete key on the keyboard. This may be a little difficult to do because the keyframes fill up all the empty areas. If this is the case, delete one keyframe and then drag select from there to select the others. After you’ve deleted all the keyframes except the last one, take the last frame and move it back to frame 0.

That’s it! Now it’s time to refine the model and prepare it for export. Make sure to return the timeline marker to 0! Note that if you find you want to add more leaves to the tree using the Crowd tool and resimulate then you will first have to select “Unbake All” from the Simulation Tools tab to be able to re-run the simulation.

12. Refinement & Export

Now let’s run the simulation. The main goal of the sim is to make sure that the leaves are behaving properly and don’t intersect with each other.

a. Refining Leaf Placement

The previous render and results of the leaf placement looks a little sparse in some areas. At this point we can use the 3D snapping tool to move some leaves around and flesh out the more sparse areas. Don’t forget you can also duplicate leaves.

b. Grouping Objects

Select all the leaves/stems in the scene using the Scene Explorer and add them to a group.

Then add the Leaves group to a new group along with the Trunk, dirt*, and pot. Remove all other objects in the scene (lights, cameras, backup geometry, etc.) and you’re ready to submit to Wayfair!

If you’d like to compare your model against ours, you can download the finished file here. Your results may vary depending on the amount of clones you used and physics results.

*For how to make dirt, see Subsection 23.

13. Thick Plastic Leaves: References & Setup

The above techniques will work great for real or fabric leaves. But for many topiaries, plastic leaves are used. All of the above approaches are the same – except for modeling the leaves and the materials.

a. Photo References

Collect some references, both up close and far away. For this part of the tutorial we’ll use SKU GSCI1369.

b. Setup

You will need to install Soulburn Scripts if you haven’t already. These plugins are super useful, and we will be using TexmapPreview quite a lot in this tutorial! To install Soulburn, go to this website and follow the instructions.

Afterwards we’ll want to make a shortcut on your toolbar. Under Customize -> Customize User Interface, go to Toolbars. Under Category, choose SoulburnScripts and scroll down until you find TexMapPreviewUI. Drag it onto a menu of your choice.

14. Thick Plastic Leaves: Modeling

The above techniques will work great for real or fabric leaves. But for many topiaries, plastic leaves are used. All of the above approaches are the same – except for modeling the leaves and the materials.

a. Create a Small Box

The largest of these leaves is probably about only 1” x .5” x .125”. Make a small box around that size. Include 4 Length Segments, 3 Width Segments, and 2 Height Segments.

b. Shape the Box

Convert the box to an edit poly and then shape it so that it is cupped and thicker in the middle, with the edges being very thin and flared out. It is important that the edges be thinner than the middle.

c. Add Stems

The source image shows that the leaves have a standardized pattern of 2-3 ribs. We need to add at least one of those patterns to our model. Do this by creating cylinders and shaping them within the Edit Poly modifier to fit within the shape of the leaf. Make sure to delete the bottom face of the cylinders.

d. Attach Stems & Assign Material Groups

Using the Attach option in the leaf object, attach the ribs to the leaf so they become a single object.

Then use the Polygon: Material ID rollout and select the leaf body. Assign it to Set ID 1 (just enter the number and press enter on the keyboard). Then select the ribs and assign them to ID #2.

e. Apply an OpenSubdiv

Apply an OpenSubdiv modifier to the leaf to test. Sometimes the subdivision can cause the stems to sink too far into the leaf body.

If this happens, go into the original Edit Poly with “Show end result on/off toggle” checked, and move the vertices until the stems are not submerged into the leaf.

Once the leaf is set up its time to move on to making a material.

15. Thick Plastic Leaves: Material

Now that we’ve built the leaf, let’s start building the material for it.

a. Lights, Camera, Action!

You will need to either use our pre-made render scene or set up your own. The most important thing is to have enough light coming from near the camera so that you can see the front of the leaf, and a light behind the leaf so that we can evaluate the 2 sided Material. The setup I am using is below – on the left is the camera view, in the middle is the render scene, and on the right is the overall scene setup. My lights are all set at a multiplier of 200 and my camera’s shutter speed and ISO are both at 100. Using the Interactive Preview Render (or IPR) will really help you with these steps.

b. Duplicating the model

To truly test our leaf, we have to see both its top, bottom, and side. To do this easily, create two instances of the leaf. An instance is an exact duplicate of an object where any changes made to the original object will also change the duplicates. Once you’ve made the duplicates, rotate them so that one has the bottom facing the camera and the other has both the top and bottom slightly visible. Bonus points if you can get the leaves to be very close to each other and overlap slightly: this will help a lot when checking the translucency next.

c. Start with the Multi/Sub-Object Material

Add a Multi/Sub-Object Material to the material editor. Set it to use two material groups and plug a material into each group. Then assign the Multi-object material to the leaves to get started.

For the first part of this tutorial we will only be editing the Slot #1 material (the leaf). We will leave the #2 material (the ribs) until last.

d. Create a VRayDirt

Create and plug a VRayDirt node into the refract map of the main material (#1). We will use this node to make the leaf transluscent around its edges but not its core – that is why it was important to make sure the leaf was thin around the edges and not the middle.

To preview the VRayDirt, click the TexmapPreviewUI button you made before. Select the leaves and make sure the VRayDirt is selected in the material editor. Under “Options” in the previewer, make sure that “Use Slate Mat Editor” is selected.

Now, when you press Apply, a render window will appear showing the selected objects with the selected material applied.

Use the texmappreview to adjust the VRayDirt’s falloff until it looks similar to below. My values are also included. For more information on VRayDirt settings, see Chaos Group’s help documentation. Make sure to check on “consider same object only”. Also ensure that you made the “Occluded” color of the VRayDirt a pure white instead of the default grey.

Note: if you don’t see anything it’s possible your VRayDirt’s radius is set too high. Lower it to something very small and try again.

e. Adjust the Refract Values

Once you’ve finished setting up the VRayDirt, do a quick render. You’ll find that the transparent areas of the leaf are actually refracting the object behind them.

To fix this, we have to go into the material and reduce the refract IOR to 1.0.

We also want to change some other items as well:

  • Make the entire object slightly transparent by changing the Refract grey value to 6.
  • Change the exit color to a bright green – the same green as the light around the edges of the leaf in the reference. Also make sure to check on “Exit Color”.
  • Change the Translucency model to Hard (wax), and the Back-side color to the same value as the exit color.
  • Changing the Thickness value won’t do much so you don’t have to worry about it generally. But if you experience an issue where your transparency isn’t showing up, give it a try.

You should start to see the transparent areas of the leaf turn a nice green color.

f. Set Up the Diffuse

There are two things to note with the diffuse of this model – that it is pretty dark and that it has a sort of dusting of white spots over it, mostly around the center. For the white spots we will use a BerconNoise and for the color we will use a VRayColor. We will use another VRayDirt to limit the spots (so they are concentrated around the center) and we will combine it with the VRayColor using a Composite node.

Start with creating a dark green VRayColor, to match the plant’s color. Then add a BerconNoise with an extremely small Scale value – I set mine to 0.01.

Then copy the VRayDirt that you used for your Refract and plug the BerconNoise into the “Unocluded” slot. This means that the areas that are not occluded will show the BerconNoise pattern, and areas that are occluded will show up as pure white.

Finally, add a Composite node. Add the VRayColor as Layer 1 and the VRayDirt as Layer 2. Then set Layer 2 to “Soft Light”.

Plug the whole thing into the Diffuse slot of the VRayMtl. Then plug the BerconNoise into the Reflect map and adjust the reflectivity of the leaf to have a tight spot and be slightly reflective (about 50-100).

Finally, copy the entire material and remove its Refract map and zero out the Refract values (set it to black). Plug the duplicate into the Material ID #2 slot. This will make sure the center ribs are a similar color to the rest of the leaf but are not transparent. Finally, set the Diffuse map value for the material at 80% and adjust the Diffuse color to be more of a brighter green. This will help it to look more natural.

If the leaf comes out too dark you can adjust it by changing the VRayColor node and the Diffuse color of the material in the #2 slot:

16. Thick Plastic Leaves: Modeling Stems with Leaves

The process here is the same as in Subsection 06: model a cylinder, taper it, distort it, attach the leaves we made to it. The steps here will only offer a brief overview of what to do, for more detail see the linked section. The most major difference is that for a fake plant like this one there are no branches – only stems with leaves.

a. Create the Stem & Assign Material/Groups

In this case I’ve decided to manually modify the stem with an edit poly, rather than adding a PathDeform modifier. Don’t forget to delete the bottom face of the cylinder!

b. Attach the Leaf to the Stem

BEFORE YOU DO THIS: remove any OpenSubdiv/Turbosmooth modifiers you applied to the leaf. Delete any additional leaves except for the original. Then within the Stem’s Edit Poly, use the Attach command to attach the leaf to the stem. Within the new object, select the stem and assign it to Material ID #2.

c. Position and Duplicate Leaves

Same as before; duplicate, position and rotate copies of the leaves. Make sure that you’re using the Selection Center as the pivot point when moving and rotating (so all the individual parts of the leaf don’t rotate separately).

d. Create Variation

Create a couple different versions of the stem.

17. Topiaries: Creating the Collision/Core Object

With topiaries, they are often made in geometric shapes – meaning spheres or cubes: simple objects. We will make a few separate objects – one for collision, then the others for distribution.

a. Create a Sphere

If you have to make the topiary a specific size, add the radius (½ diameter) of the sphere to the height of the tallest stem, then multiply it by 2. This will give you the diameter of the topiary you’re about to make. For example a sphere with a diameter of 12” will have a radius of 6” + tallest stem height of 4” = 10” radius * 2 = 20” diameter topiary. Make sure to name it something obious, like TopiaryCore.

b. Apply a Dark Green Material to It

The Sphere will only really be there to add some mass to the center of the topiary. Make sure it fades away as much as possible by removing any RGloss (completely black value) and making it a very very dark material.

c. Make it a Static Rigid Body

You will need to open the MassFX toolbar.

To enable it, right-click on an empty area of the menu and find it. Once you click it it will appear as a floating window, which you can dock on your toolbar for ease-of-use by clicking and dragging the top of the window onto an empty area of your toolbar.

Once you do this, you will need to set the sphere as a Static Rigid Body. Since the object is a standard primitive the default Convex shape is fine. Just bump up the vertice count so the collision mesh has enough vertices to completely enclose the sphere. For more details on these processes see Subsection 9.

18. Topiaries: Creating the Distribution Objects

Now we will be creating some distribution objects. These will only be used for distributing the leaves. The reason we have to make several separate objects is to make sure that the leaves actually stick outwards.

a. Copy the Core Object

We will be using the object we made before as a baseline. Make four copies of it, leaving the original and remove the MassFX Rigid Body modifier from each. Note that below the far right model is the original with its Rigid Body Modifier intact, while the other models are all copies with no Rigid Body modifier.

b. Split the Objects into Quadrants

Take the models and delete vertices so that the models are split into four separate interlocking parts.

c. Align the Objects With the Core Object

Take each individual quadrant and use the Align tool (Alt-A) to align it with the Core Object. Use X/Y/Z and set it to Pivot/Pivot (assuming you did not change the pivot point of the quadrant models). The objects in the image to the right have had different materials assigned to them so that you understand the sphere is now made of four separate object “chunks”. You do not need to change their colors.

19. Thick Plastic Leaves: Preparing the Leaves & Stems for Duplication & Distribution

The process for preparing the stems here is the same as it is in Subsection 9. Follow those steps to get the results below. These models are more complex than the previous ones so you may need to reduce the mesh settings – Mesh Detail, Hull Size, Max Verts. Note: It’s a good idea to save before generating a mesh in case Max crashes.

You should end up with a group for each leaf variant, where the leaf (with collision mesh) is grouped with its ball & socket constraint and the group’s pivot point is at the bottom of the leaf, where the constraint is.

20. Thick Plastic Leaves: Distributing the Leaves and Stems

The process here is the same as in Subsection 10, except for the Rotation tab. See the steps below for the differences.

a. Use the Crowd Tool to Distribute the Leaves on Each Object

For the top sphere (in this image, red) you will not have to change anything.

b. Re-Orient Distribution Items

For the other quadrants of the sphere, you will need to change the “Up-Axis” under the Rotation tab. This will allow you to pattern the leaves around the sphere but also have them stick out away from it (and not into it). Note that for the “belt” quadrants you should use a lower density than you use for the top and bottom quadrants.

For the bottom Sphere (in this case, the yellow one), you will need to keep the same Up Axis but change the positive/negative direction of the axis by clicking on the blue plus sign to turn it into a black minus sign.

Don’t forget to distribute the different variations of each leaf to each quadrant of the sphere!

Also note that after you’ve distributed an object and its clones, the source object will also be moved to the distribution surface. Just select it from the list view when it comes time to distribute it on another quadrant. You will want to make sure that the sphere is as covered as possible.

21. Thick Plastic Leaves: Simulate

The process here is the same as in Subsection 11. The only difference is that you will need to hide or delete the sphere quadrants and show the original TopiaryCore object, which contains the collision mesh. This mesh will keep the topiary leaves from being pushed inside the topiary (pointing inward towards the center). For this simulation it is especially recommended to use No Gravity. Note: Very complex collision meshes will dramatically increase simulation time. Make sure to adjust Substeps and Solver Iterations to compensate.

22. Thick Plastic Leaves: Refinement & Export

The process here is the same as in Subsection 12. You may choose to keep or discard the distribution objects you created (the quarters of the sphere). The only difference is that you should take the original core object and scale it down slightly so that it blocks some of the light from passing through the leaf clusters, to give the impression of a fuller volume. You can download a final version of the model below here, but note that it was made with a different distribution mesh than the one shown above.

23. Creating Dirt

Creating dirt for a plant model is a simple process.

a. Create a Plane and Shape It

Shape a plane to the limits of the containment vessel – for example, a pot.

b. Add a TurboSmooth

Add some extra geometry that we can work with by adding a TurboSmooth to the plane. Add at least 2 iterations.

c. Freeze the TurboSmooth

Add an Edit Poly modifier above the TurboSmooth and Collapse it to “freeze” the extra TurboSmooth geometry.

d. Apply a Planar Map

Apply a Planar UVW Map to the object.

e. Sculpt the Dirt

With the Editable Poly modifier selected, go to the FreeForm tab on the Graphite modeling Tools. Select the Push/Pull tool and adjust the settings in the pop-up window. Size will change the size of the brush, Strength will change the rate of change, and Offset will change how much the change is. Use this tool to give the dirt some height variation.

When using the brush tool, Left Click will push the geometry up to sculpt it. Holding Shift + Left click will smooth the geometry out, and holding Ctrl + Left Click will push the geometry down.

f. Create a Material is a great resource for finding dirt textures. They even come with images you can use for displacement! Just make sure to put them in a VRayDisplacement modifier.