Finely grate the zest of 1 medium orange (about 2 teaspoons) and 1 medium lemon (about 1 teaspoon) into a medium saucepan.

Add 1 quart whole milk, 1/2 cup granulated sugar, and 1 (3-inch) cinnamon stick. Bring to a boil over medium heat. Remove the pan from the heat and add 1/3 cup dry sherry. Let sit until cool enough to handle. Meanwhile, cut 1 (12-ounce) day-old baguette crosswise into 1-inch-thick slices (about 16). Arrange in a single layer on a rimmed baking sheet.

Pour the milk mixture over the bread and let it soak for 5 minutes. Flip each slice of bread. Let the bread sit until it absorbs most of the liquid but isn't falling apart, about 5 minutes more.

Heat 1 cup olive oil in a large frying pan over medium-high heat until shimmering. Meanwhile, whisk 4 eggs together in a medium bowl.

Working with 1 slice at a time, dip the bread in the egg mixture to coat. Let the excess drip off, then add to the pan. Repeat with as many slices as needed until the pan is filled with a single layer. Fry until golden-brown on the bottom, about 4 minutes. Flip the slices and fry until the second side until golden-brown, 3 to 4 minutes more. Transfer to a paper towel-lined plate or baking sheet. Repeat dipping and frying the remaining bread. Drizzle with honey to serve.