What's the best way to merge to arrays based on key values in each array?

I have these two arrays that I am currently using two loops to merge. I was wondering if there was a better way to merge these arrays:

$args = array(
  'Name' => 'Test Name', 
  'Email' => '[email protected]', 
  'Message' => 'There is no place like 127.0.0.1!'
);

$fields = array(
  2 => array(
    'label' => 'Name',
    'id' => 2
  ),
  3 => array(
    'label' => 'Email',
    'id' => 3
  ),
  4 => array(
    'label' => 'Message',
    'id' => 4
  )
);

I am trying to merge them into an array like this:

$merged = array(
  2 => array(
    'label' => 'Name',
    'id' => 2,
    'value' => 'Test Name'
  ),
  3 => array(
    'label' => 'Email',
    'id' => 3,
    'value' => '[email protected]'
  ),
  4 => array(
    'label' => 'Message',
    'id' => 4,
    'value' => 'There is no place like 127.0.0.1!'
  )
);

I am currently using this dual foreach loop to do it:

foreach ( $args['fields'] as $k => $v ) {
        foreach( $fields['fields'] as $i => $item ) {
            if ( $item['label'] === $k ) {
                $fields['fields'][$i]['value'] = $v;
            }
        }
    }

I have tried using array_merge and array_merge_recurisve, but it throws off the array keys from the $fields array.

Any thoughts or better approaches to try and achieve my desired outcome?