The simplest and most common sweetener in homemade liqueurs is a syrup of sugar made from 2 parts sugar completely dissolved in one part water. In most every book I've run across, it's the same recipe. But there are alternatives you can use, and they can have dramatic and unexpected effects on your finished product. Some recipes specify alternative sweetening, like the Tart Apple recipe I use that specifies that dry sugar be added to the vodka directly. Some recipes I have mention some of the alternatives as variations on the theme, like brown sugar instead of white. But in reality, any recipe can use any of the available sweeteners. You'll have to experiment to determine the right replacement proportion, but it's often worth the effort.
So the first obvious sweetener after sugar is brown sugar. When I first started out, I just substituted light brown sugar for the same measure of white sugar, and dissolved away. The results, I thought, were pleasant. What makes brown sugar brown (apologies to people who already know) is molasses, and that can give the resulting liqueur a heartier and heavier flavor, even bordering on "smoky." Use your own judgment for what flavors might benefit or not. The Tart Apple mentioned above actually worked quite well substituting the same measure of dry brown sugar for the white before steeping. It made for a nice Autumn drink.
Since then, I've been told that one should use less brown sugar than white because of the molasses, and yet I've also seen the exact opposite. I've also seen variations which use some measure of white and brown sugar together. And finally, I've always been tempted to just go over the edge and substitute dark brown sugar, but I expect it would just dominate the taste instead of bolstering it. But someday, I'll know for myself.
Beyond sugars, though, there is a whole world of natural sweeteners. Honey is one of the more popular ones, and offers a wide range of subtle variations. A good starting point is simply substituting the same measure of a generic honey for your sugar syrup, and then vary to taste. But from there, consider using specific kinds of honey. Orange blossom honey can be very different from clover, and other flowers each can add their own special character to your liqueurs. Remember that some liqueurs are made from nothing but flowers, syrup and vodka.
Honey offers a special problem in filtration, however, being prone to leaving very fine mists at the bottom of your jar. You can try to filter them out using very fine filters, but you may just find that decanting or racking (siphoning) to be an easier approach. You may lose a few extra ounces of liqueur, but you'll save yourself a lot of time. I've also heard mead-makers talking about boiling honey to get the wax to rise to the surface to be skimmed away, so it won't make trouble for you further down the road. I don't know what the heat would do to the honey's natural flavor, though.
Beyond these, though, many things could be used to some effect. Corn syrup, natural cane sugar and such things can all be used. Get weird. Remember, without people dreaming up weird things to do with normal foods, we wouldn't have cheese or mayonnaise. Or sausage.
Once you've found the taste you like, you may find the level of sweetness to be unpleasant. I personally prefer my liqueurs not to be too sweet. At the same time, I don't like them too bitter or dry either, so I'm left with a delicate balancing act trying to find the right taste. The absolute minimum sweetening for any liqueur is what it takes to mask the vodka. That will depend entirely on what else is going into the recipe: cranberries could mask cod liver oil without breaking a sweat, but raspberries need all the help they can get. If you're really disappointed about the taste of something you've made, take a look at the sweetening ratio first. Along the way, you may also find that increasing the amount of flavorings per unit of vodka helps the equation, too. I certainly find that true fairly often.
Once you know how little sweetening it requires, though, then you have to find out how much you want. I personally would take everything at the minimum, if only I had time to determine what it was. But most of my friends (the people who drink the majority of what I make) like things a little sweeter than that. So you need to try to find a happy medium. After a few iterations of this, you'll probably be saddened to realize that, by and large, you're just approaching the original recipes again, but for some flavors it's well worth the odyssey. Of course, no one in their right mind is going to make all these little experiements, but sometimes you should just try a small batch with a completely off-the-wall sweetening. If it works, use it as the standard recipe, if it doesn't, no harm done.
So there are some standard formulas you'll want to remember.
Finally, a note on that recipe for standard syrup (also called sugar syrup, and simple syrup). It actually yields about 10% more syrup than you thought you were getting. Thus, 1 cup of sugar and 1/2 cup of water together yield about 1.1 cups of syrup. Some recipes, when they say 1 cup of syrup, mean "the 1.1 cups you would get from using the standard recipe." Others mean precisely, "1 cup of simple syrup, and throw away the excess." But rest assured no recipe will ever tell you what it was expecting. You have to figure it out. Still, that's part of the joy of homemade liqueurs - figuring it out yourself. If you have only one shot to make a liqueur, err on the side of caution, and add extra sweetening as needed later. You can't subtract sweetening, only add more. And it's always a matter of taste anyway.