Preheat the oven to 350°F. Grease three 6-inch cake pans and line the bottoms with parchment paper. Set aside.
Along with ¼ cup of sugar, grind the lavender until small pieces using a spice grinder, mini food processor, or mortar and pestle.
In a large mixing bowl, whisk together the lavender sugar, flour, remaining plain sugar, baking powder, and salt.
In a separate bowl or large measuring cup, whisk together the milk and eggs. Pour in the butter, oil, and vanilla.
Using a hand mixer or whisk, blend the wet ingredients into the dry ingredients. Finish by folding together with a rubber spatula until smooth.
Pour the mixture into the prepared pans. Bake for 25 to 30 minutes or until a toothpick inserted comes out clean or with a few moist crumbs. Completely cool the cake before frosting.