With the mixer on medium speed, stream in the olive oil. Once all the olive oil is added, increase the speed to high. With the mixer on low speed, stream in the milk. Stir in the dry ingredients into the wet until just combined.
Pour the cake batter into the prepared pan. Bake the olive oil cake at 350 F (177 C) for 40-45 minutes. Cool the cake in the cake pan placed on a cooling rack for 20 minutes. Remove the cake to cool to room temperature on the cooling rack.
Extract all the juice from the grapefruits. Pour the juice into a small sauce pan. Heat the juice over medium high heat. Once the juice has reduced down to 2-3 tablespoons (30-45 ml) remove the pan from the heat. Whisk into the powered sugar.