Firefox scale image disappears

This sounds like a bug indeed, particularly with the text-overflow property.

You are setting the wrapper's <a> element's font-size to 0, you also say that text should not overflow and that overflowing text content should get replaced by an ellipsis …. When you apply the scale(1.1), the inner <img> element which is perceived as text because of the display: inline rule does overflow. And that's where the text-overflow: ellipsis algorithm kicks in and makes your image disappear:

a {
  font-size: 0px;
  overflow: hidden;
  text-overflow: ellipsis;
  display: block;
  width: 120px;
  height: 120px;
}
a > img {
  display: inline;
  transition: transform .3s linear;
  max-width: 100%;
  max-height: 100%;
}
a > img:hover {
  transform: scale(0.9);
}
a.no-ellipsis {
  text-overflow: clip;
}

ellispis:
<a>
  <img src="https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png">
</a>
no ellipsis:
<a class="no-ellipsis">
  <img src="https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png">
</a>

To workaround that issue, you have several ways, including the one in the snippet (text-overflow: clip), or displaying your <img> as block, or setting your wrapper's font-size to 1px, and probably others...