Set matplotlib font by family name when there are multiple fonts with same name

How can I tell matplotlib to use a specific variant of a font when they both have the same name and characteristics?

For example, I can tell matplotlib to use Latin Modern Roman:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import numpy as np
from pathlib import Path

import matplotlib as mpl
import PyQt5
mpl.use('Qt5Agg')
import matplotlib.pyplot as plt

mpl.rcParams['font.family'] = 'Latin Modern Roman'
mpl.rcParams['font.weight'] = '400'
mpl.rcParams['font.style'] = 'normal'
mpl.rcParams['font.variant'] = 'normal'
mpl.rcParams['mathtext.fontset'] = 'custom'
mpl.rcParams['mathtext.default'] = 'it'
mpl.rcParams['mathtext.rm'] = 'Latin Modern Roman:normal'
mpl.rcParams['mathtext.it'] = 'Latin Modern Roman:italic'
mpl.rcParams['mathtext.bf'] = 'Latin Modern Roman:bold'

mpl.rcParams['xtick.labelsize'] = 8
mpl.rcParams['ytick.labelsize'] = 8
mpl.rcParams['axes.titlesize'] = 10
mpl.rcParams['axes.labelsize'] = 10

x = np.linspace(0,2*np.pi,100)
y = np.sin(x)

fig1 = plt.figure(figsize=(3, 3*(9/16)), dpi=300)
ax1 = fig1.gca()
ax1.plot(x, y, c='r', linewidth=1.0, zorder=20)
ax1.set_xlabel(r'$x\ label$')
ax1.set_ylabel(r'$y\ label$')
fig1.tight_layout(pad=0.15)
plt.show()

OK, that's nice. But what if I want to use a specific "substyle" of Latin Modern Roman? For example, on my system, when I list the available font entries having name='Latin Modern Roman' and weight=400 and style='normal' and variant='normal' through:

fonts = mpl.font_manager.fontManager.ttflist
for f in fonts:
    if (f.name=='Latin Modern Roman'):
        if all([(f.style=='normal'),(f.weight==400),(f.style=='normal'),(f.variant=='normal')]):
            print(f.name)
            print(Path(f.fname).stem)
            print(f.fname)
            print('weight : %s'%str(f.weight))
            print('style : %s'%str(f.style))
            print('stretch : %s'%str(f.stretch))
            print('variant : %s'%str(f.variant))
            print('\n')

I get:

Latin Modern Roman
lmroman9-regular
/usr/share/texmf/fonts/opentype/public/lm/lmroman9-regular.otf
weight : 400
style : normal
stretch : normal
variant : normal

...
...

Latin Modern Roman
lmroman10-regular
/usr/share/texmf/fonts/opentype/public/lm/lmroman10-regular.otf
weight : 400
style : normal
stretch : normal
variant : normal

So how would I tell matplotlib that I want to use lmroman9-regular.otf vs lmroman10-regular.otf? In mpl.rcParams I can only specify font.family,font.weight,font.style and font.variant so if the two .otf files have all the same values, how can I tell it to use one .otf rather than the other? There are actually differences between the fonts, for example:

I've tried referencing the .otf file directly with a matplotlib.FontProperties instance, then renaming it, then pointing to the new name like:

prop = mpl.font_manager.FontProperties(fname='/usr/share/fonts/opentype/lmodern/lmroman10-regular.otf')
prop.set_name('Latin Modern Roman Type 10')
mpl.rcParams['font.family'] = 'Latin Modern Roman Type 10'

but then I get findfont: Font family ['Latin Modern Roman Type 10'] not found. Falling back to DejaVu Sans. because prop.get_name() still (mysteriously) returns Latin Modern Roman and not Latin Modern Roman Type 10

How can I tell matplotlib that I want to use a one over the other?

It's possible to explicitly use fontproperties to point to a font_manager.FontProperties() instance in most set_label and set_ticklabel calls (like in this answer), but I'm looking for a way to set this globally through rcParams.

Note: I would NOT like to use LaTeX rendering (text.usetex) in matplotlib.

System:

• Ubuntu-20.04 on WSL2 on Windows 10 Pro 19042
• Python 3.8.5 / [GCC 9.3.0] on linux
• matplotlib 3.4.1