How do I specify which driver is used for a device when more than one are applicable

I'm working on getting a more modern Linux working on a single board computer where the original company has gone under but the hardware is well supported by the open source community. I've made very good progress but I'm running into an issue where the incorrect driver module is loaded for a particular device.

The platform is an Allwinner R8 (Which is the same as the Allwinner A13) and the device is the resistive touchscreen. Distribution is Slackware-current for ARM, which means no systemd but it does use eudev. I've compiled kernel 5.2.0-rc6 from source because I wanted to try the Mali lima driver and needed some modules not included in the distro build anyway. The relevant sections from the device tree are as follows.

u-boot/arch/arm/dts/sun5i.dtsi:

587         rtp: rtp@1c25000 {                                                                                                                                                                 
588             compatible = "allwinner,sun5i-a13-ts";                                                                                                                                         
589             reg = <0x01c25000 0x100>;                                                                                                                                                      
590             interrupts = <29>;                                                                                                                                                             
591             #thermal-sensor-cells = <0>;                                                                                                                                                   
592         };

overlay (which is cribbed from the manufacturer's overlay for a modified 4.4 Kernel included out of tree patches for some devices)

 76   /* Enable the touchscreen */                                                                                                                                                             
 77   fragment@4 {                                                                                                                                                                             
 78     target = <&rtp>;                                                                                                                                                                       
 79     __overlay__ {                                                                                                                                                                          
 80       touchscreen-inverted-x;                                                                                                                                                              
 81       touchscreen-inverted-y;                                                                                                                                                              
 82       allwinner,ts-attached;                                                                                                                                                               
 83     };                                                                                                                                                                                     
 84   };

The problem is that the sun4i-gpadc module is loaded for this device when what I need is the sun4i-ts. The former is the general purpose ADC driver while the latter is the touchscreen driver that I need.

root@pocketslack:/etc# udevadm info -a /sys/devices/platform/soc\@1c00000/1c25000.rtp

  looking at device '/devices/platform/soc@1c00000/1c25000.rtp':
    KERNEL=="1c25000.rtp"
    SUBSYSTEM=="platform"
    DRIVER=="sun4i-gpadc"
    ATTR{driver_override}=="(null)"

If I use modprobe to remove sun4i_gpadc and sun4i_ts then use it to load them starting with sun4i-ts first, it grabs the device and it works properly.

root@pocketslack:/etc# modprobe -r sun4i_gpadc
root@pocketslack:/etc# modprobe -r sun4i_ts
root@pocketslack:/etc# modprobe sun4i_ts
root@pocketslack:/etc# modprobe sun4i_gpadc
root@pocketslack:/etc# udevadm info -a /sys/devices/platform/soc\@1c00000/1c25000.rtp

  looking at device '/devices/platform/soc@1c00000/1c25000.rtp':
    KERNEL=="1c25000.rtp"
    SUBSYSTEM=="platform"
    DRIVER=="sun4i-ts"
    ATTR{driver_override}=="(null)"

I have a udev rule which I also got from the company's old repo which I thought would handle this, but the device is still taken by the gpadc driver.

root@pocketslack:/etc# cat /etc/udev/rules.d/10-sun4i-ts.rules 
ACTION=="add", SUBSYSTEM=="input", DRIVERS=="sun4i-ts", SYMLINK+="input/sun4i-ts-%k"

I'll admit to being totally new at udev though so I'm not sure exactly what that rule is telling it to do. My hunch is it's just telling it to load that module and how to name any device it attaches to. I'm also not sure I need the sun4i-gpadc driver at all, but I gather from the kernel documentation it seems to be needed for the SoC's thermal sensors to function. This device also has pins exposed for projects, so I thought it may also be useful to have in case one actually wanted to use the included ADC that way. It also seems like there should be some mechanism to specify the correct driver when several could apply and I figured I should just learn how to do that rather than just not build or blacklist the gpadc module.