Why `audioPlayerDidFinishPlaying` is still alive after dismiss the current viewController it belongs to

In playSong you assign self as the delegate to AudioPlayer, so it retains a reference to your PlayerViewController, so even though you have dismissed it, the AudioPlayer keeps it alive, and calls audioPlayerDidFinishPlaying when it finishes playing the song.

Of course, the next natural question is, "why does the AudioPlayer remain alive?" I'm pretty sure that's because behind the scenes it does the work of playing the song in an @escaping closure in a concurrent task. To do its work that closure has to capture a reference to the player, keeping it alive.

Regarding getting duplicate PlayerViewController instances, that's because the AudioPlayer is a singleton and the PlayerViewController is being used as the AudioPlayer's delegate, so the PlayerViewController is kept alive by the AudioPlayer. Opening the view again spawns a new PlayerViewController, but the old one still exists in the AudioPlayer.

The solution is to refactor so PlayerViewController is no longer the AVAudioPlayerDelegate.

So you move the methods for AVAudioPlayerDelegate conformance from your PlayerViewController to a separate object. In the following code I assume songs is an array of Song, and that modelController is a ModelController, since those definitions aren't in the provided code.

In this case, I think the best candidate for the AVAudioPlayerDelegate seems to be AudioPlayer itself.

Also since we're trying to decouple the AudioPlayer from the PlayerViewController, we still need a way for the AudioPlayer to communicate when a song ends so that it can update it's time label. In preparation for that I declare a PlayerListener protocol (where in this case "listener" refers to the design pattern rather than listener of music).

protocol PlayerListener: class
{
    func receivePlayerMessage(_ message: AudioPlayer.Message)
}

@objc class AudioPlayer: NSObject, AVAudioPlayerDelegate
{
    // Singleton pattern
    static let shared = AudioPlayer()
    var player: AVAudioPlayer? = nil

    var songs: [Song] = []
    var position: Int = 0
    var modelController: ModelController? = nil
    
    enum Message
    {
        case songFinished
        case songStarted
    }
    
    private var listeners: [PlayerListener] = []
    
    private override init() {}
    
    func add(listener: PlayerListener) {
        listeners.append(listener)
    }
    
    func remove(listener: PlayerListener) {
        listeners.removeAll { $0 === listener }
    }
    
    func broadcast(message: Message) {
        listeners.forEach { $0.receivePlayerMessage(message) }
    }
    
    func audioPlayerDidFinishPlaying(_ player: AVAudioPlayer, successfully flag: Bool)
    {
        guard !flag else {
            fatalError("the audio doesn't finished playing successfully.")
        }
        
        goNextTrack()
        print("I'm alive")
        broadcast(message: .songFinished)
    }
    
    func goNextTrack()
    {
        guard let playMode = modelController?.playMode else { return }
        
        switch playMode
        {
            case .cycle: position = (position + 1) % songs.count
            case .cycleOne:  break
            default: songs.shuffle()
        }
        
        playSong()
    }
    
    func playSong()
    {
        guard songs.indices.contains(position) else { return }
        
        let song = songs[position]
        guard let url = song.path else { return }
        
        AudioPlayer.shared.play(url: url)
    }
    
    func play(url: URL)
    {
        do
        {
            player = try AVAudioPlayer(contentsOf: url)
            guard let player = player else { return }
            player.delegate = self
            player.prepareToPlay()
            player.play()
            broadcast(message: .songStarted)
        }
        catch { print(error) }
    }
}

Remove AVAudioPlayerDelegate conformance from your PlayerViewController, and add conformance to PlayerListener.

Presumably it still needs to respond to playSong() and maybe goNextTrack() in response to UI elements. Also since the AudioPlayer needs song data, you need to set that whenever your view loads/appears. Because you mention re-using the view controller, let's do that in viewDidAppear(), though I question that approach - in my mind each view (or maybe view hierarchy) should have its own view controller instance. But that's just me.

// in PlayerViewController
    var position: Int
    {
        get { AudioPlayer.shared.position }
        set { AudioPlayer.shared.position = newValue }
    }
    
    func receivePlayerMessage(_ message: AudioPlayer.Message)
    {
        switch message
        {
            case .songStarted: updateTimeLabel()
            default: break
        }
    }

    override func viewDidAppear()
    {
        AudioPlayer.shared.songs = songs
        AudioPlayer.shared.modelController = modelController
        AudioPlayer.shared.add(listener: self)
    }
    
    override func viewDidDisappear() {
        AudioPlayer.shared.remove(listener: self)
    }

    func goNextTrack() {
        AudioPlayer.shared.goNextTrack()
    }

    func playSong() {
        AudioPlayer.shared.playSong()
    }

Now you have one player delegate for the life of the AudioPlayer. When your view controller is dismissed it will not be kept alive (at least not by the AudioPlayer, and thus can be deinitialized and deallocated, so you don't end up with multiple instances of the view controller.

You did mention that you wanted to re-use the view controller, which is why we set AudioPlayer data in viewDidAppear instead of viewDidLoad. I'm not sure that's a good idea, and is a separate question. I'll address it somewhat in comments.