nmcli: limit number of active connections

That's not how Ethernet works; there's no "connections" in ethernet; that's a concept from two layers higher.

So, you could try to make a firewall behave in a way that limits specific kind of activity, e.g. the ability to send TCP/IP packets, to a certain number of IP addresses. This would probably mean writing a daemon or BPF script beyond my experience in the Linux networking stack, to monitor the number of clients.

But: What is the number of clients?

• IP addresses? Do you mean IPv4 or IPv6? Everyone can just pick as many IP addresses as they like for themselves, and matter of fact, a lot of system services rely on temporary or auto-configured or default addresses. This is neither a good protection (I could steal the IP address of someone who currently has "allowance"), nor is it even superficially save against denial of service (2 lines of shell code give me a 10000 IP addresses on my interface, good night, neighbor!).
• Ethernet MAC addresses? Same problem as IP, these can be picked arbitrarily, and especially in environment with moving equipment/wireless access, these are often randomized
• Established TCP/IP connections to the service: now we're talking. Your service is itself in control of this, and could simply reject connections under circumstances that seem to be clear only to the service itself!

In all honesty, this sounds like an application-level problem you're trying to solve at some deeper layer in the network. But the deeper layers of the network were never meant for access control. Simply don't. Whatever the service is your clients access, make it have authentication and a notion of a session – and simply reject if the maximum number of sessions is exhausted.