In a large mixing bowl, cream the sugar and butter together until light and fluffy. This takes about 2-3 minutes on medium speed. On low, mix in the eggs, vanilla extract and sour cream until well combined.
Alternating dry ingredients and milk, mix in ½ of the dry ingredients into the wet, followed by the milk and the remaining dry ingredients.Fold in 1 ½ cups of the brown sugar cinnamon apples until just combined.
Cool the cake in the loaf pan for 15 minutes. In a mixing bowl, whisk the powdered sugar with vanilla bean paste and milk to your desired consistency. Drizzle over the cooled apple fritter loaf, slice and enjoy!