ImmutableHashSet .Contains returns false

I have a list (to be precise ImmutableHashSet<ListItem> from System.Collections.Immutable) of base items and try to call the following code

_baseList.Contains(derivedItem)

but this returns false.

Even though the following code lines all return true

object.ReferenceEquals(_baseList.First(), derivedItem)
object.Equals(_baseList.First(), derivedItem)
_baseList.First().GetHashCode() == derivedItem.GetHashCode()

I can even write the following and it returns true:

_baseList.OfType<DerivedClass>().Contains(derivedItem)

What am I doing wrong, I would like to avoid writing the .OfType stuff.

Edit:

private ImmutableHashSet<BaseClass> _baseList;

public class BaseClass
{

}

public class DerivedClass : BaseClass
{

}

public void DoStuff()
{
    var items = _baseList.OfType<DerivedClass>().ToList();
    foreach (var derivedItem in items)
    {
        RemoveItem(derivedItem);
    }
}

public void RemoveItem(BaseClass derivedItem)
{
    if (_baseList.Contains(derivedItem))
    {
        //doesn't reach this place, since _baseList.Contains(derivedItem) returns false...
        _baseList = _baseList.Remove(derivedItem);
    }

    //object.ReferenceEquals(_baseList.First(), derivedItem) == true
    //object.Equals(_baseList.First(), derivedItem) == true
    //_baseList.First().GetHashCode() == derivedItem.GetHashCode() == true
    //_baseList.OfType<DerivedClass>().Contains(derivedItem) == true
}

Edit2:

Here a reproducible code of my problem, seems like ImmutableHashSet<> caches GetHashCode and doesn't compare the current GetHashCode with the entries inside the list, is there a way to tell ImmutableHashSet<> that the GetHashCode of the items could be different, atleast for the item I am currently checking since hey its the damn same reference...

namespace ConsoleApplication1
{
    class Program
    {
        private static ImmutableHashSet<BaseClass> _baseList;

        static void Main(string[] args)
        {
            _baseList = ImmutableHashSet.Create<BaseClass>();
            _baseList = _baseList.Add(new DerivedClass("B1"));
            _baseList = _baseList.Add(new DerivedClass("B2"));
            _baseList = _baseList.Add(new DerivedClass("B3"));
            _baseList = _baseList.Add(new DerivedClass("B4"));
            _baseList = _baseList.Add(new DerivedClass("B5"));

            DoStuff();
            Console.WriteLine(_baseList.Count); //output is 5 - put it should be 0...
            Console.ReadLine();
        }

        private static void DoStuff()
        {
            var items = _baseList.OfType<DerivedClass>().ToList();
            foreach (var derivedItem in items)
            {
                derivedItem.BaseString += "Change...";
                RemoveItem(derivedItem);
            }
        }

        private static void RemoveItem(BaseClass derivedItem)
        {
            if (_baseList.Contains(derivedItem))
            {
                _baseList = _baseList.Remove(derivedItem);
            }
        }
    }

    public abstract class BaseClass
    {
        private string _baseString;
        public string BaseString
        {
            get { return _baseString; }
            set { _baseString = value; }
        }

        public BaseClass(string baseString)
        {
            _baseString = baseString;
        }

        public override int GetHashCode()
        {
            unchecked
            {
                int hashCode = (_baseString != null ? _baseString.GetHashCode() : 0);
                return hashCode;
            }
        }
    }
    public class DerivedClass : BaseClass
    {
        public DerivedClass(string baseString)
            : base(baseString)
        {

        }
    }
}

If I would change the ImmutableHashSet<> to ImmutableList<> the code works fine, so if you guys don't come up with any good idea I will switch to the list.