How to reduce size of scrollbar?

The ListBox uses a ScrollViewer internally in its control template. The default ScrollViewer template will use the scroll bar width defined in the SystemParameters. A simple way to adapt it is to override the key in the ListBox resources. Please note that there are different keys for the horizontal an vertical scroll bars. You might want to adapt both in the long run.

<ListBox ItemsSource="{Binding Path=Items}" ScrollViewer.CanContentScroll="False">
    <ListBox.Resources>
        <system:Double x:Key="{x:Static SystemParameters.VerticalScrollBarWidthKey}">4</system:Double>
        <Style TargetType="ScrollBar">
            <Setter Property="Width" Value="4"/>
            <Setter Property="Template" Value="{DynamicResource MyScrollBar}"/>
        </Style>
    </ListBox.Resources>
</ListBox>

Another way is to create a custom control template for the ScrollViewer. You can refer to the official styles and template documentation for ScrollViewer to get an idea how to build this template. There is already an example, where you can adapt the scroll bar width.

<Style x:Key="MyScrollViewer"
       TargetType="{x:Type ScrollViewer}">
  <Setter Property="OverridesDefaultStyle" Value="True" />
  <Setter Property="Template">
    <Setter.Value>
      <ControlTemplate TargetType="{x:Type ScrollViewer}">
        <Grid>
          <Grid.ColumnDefinitions>
            <ColumnDefinition Width="Auto" />
            <ColumnDefinition />
          </Grid.ColumnDefinitions>
          <Grid.RowDefinitions>
            <RowDefinition />
            <RowDefinition Height="Auto" />
          </Grid.RowDefinitions>
          <Border Grid.Column="1"
                  BorderThickness="0,1,1,1">
            <Border.BorderBrush>
              <SolidColorBrush Color="{DynamicResource BorderMediumColor}" />
            </Border.BorderBrush>
            <ScrollContentPresenter CanContentScroll="{TemplateBinding CanContentScroll}" />
          </Border>
          <ScrollBar x:Name="PART_VerticalScrollBar"
                     Value="{TemplateBinding VerticalOffset}"
                     Maximum="{TemplateBinding ScrollableHeight}"
                     ViewportSize="{TemplateBinding ViewportHeight}"
                     Visibility="{TemplateBinding ComputedVerticalScrollBarVisibility}"
                     Width="4"/>
          <ScrollBar x:Name="PART_HorizontalScrollBar"
                     Orientation="Horizontal"
                     Grid.Row="1"
                     Grid.Column="1"
                     Value="{TemplateBinding HorizontalOffset}"
                     Maximum="{TemplateBinding ScrollableWidth}"
                     ViewportSize="{TemplateBinding ViewportWidth}"
                     Visibility="{TemplateBinding ComputedHorizontalScrollBarVisibility}"/>

        </Grid>
      </ControlTemplate>
    </Setter.Value>
  </Setter>
</Style>

You can either make this style implicit by omitting the x:Key, so it will be applied to all ScrollViewers in scope or you can create a custom ListBox control template, where you sepcify this template for the internal ScrollViewer in order to apply it only there.