How to remove the legend key border in ggplot's geom_sf?

By default geom_sf draws a legend whose keys (in the fill color) are surrounded by a frame in the border color (i.e., the color aesethetic). Is there a way to remove this frame? Apparently, this frame is inside the key rectangle so it cannot be overwritten by modifying legend.key in the example below.

library(tidyverse)
library(sf)
library(spData)

spData::world %>% 
  ggplot(aes(fill = continent)) +
  geom_sf() +
  theme(legend.key = element_rect(colour = "red"))

^{Created on 2022-08-11 by the reprex package (v2.0.1)}

Ideally, I would not only remove the frames but also remvoe the vertical space between the legend keys entirely, similar to the appearance of guide_colorsteps.

My hacky attempt here succeeds in removing the frames but the rectangles are not perfectly adjacent yet. Is there a) a less hacky way to remove the frames, and b) a way to remove the vertical space between the rectangles altogether?

library(tidyverse)
library(sf)
library(spData)

spData::world %>% 
  ggplot() +
  geom_sf(aes(fill = continent, color = continent)) +
  geom_sf(data = st_geometry(spData::world), fill = NA, color = "black") + 
  theme(legend.spacing.y = unit(0, 'cm')) +
  guides(fill = guide_legend(byrow = TRUE))

^{Created on 2022-08-11 by the reprex package (v2.0.1)}