Handle Navigation Back Button

I have two fragments, A and B. Fragment A launches B.

findNavController().navigate(R.id.action_A_to_B)

Now in Fragment B I have a button on the UI to return to Fragment A. How should I do this using the NavController? Should I just create an Action in my navigation? Or can I just do something that will return to the caller?

I ask because adding an Action means that A -> B and B -> A in the graph looks kinda weird.