Bytebuddy Java Agent: AgentBuilder affects control flow of a second, unrelated AgentBuilder

So I am implementing a Java Agent that installs multiple AgentBuilder, one of them only if an environment variable was defined correctly. The problem I face is that the behavior of a second AgentBuilder changes depending on whether the optional one was installed. To me it seems like installing the optional AgentBuilder affects which methods, that are instrumented by the second unrelated AgentBuilder, get called and how often.

if (EnvVar.isTrue()) {
            // These methods are customs so I don't have to rewrite the whole AgentBuilder-chain...
            getAgentBuilder1(inst, tempFolder).installOn(inst);
        }

getAgentBuilder2(inst, tempFolder).installOn(inst);

The AgentBuilder1 uses the following type matching:

ElementMatchers.isSubTypeOf(InputStream.class)
   .and(
        ElementMatchers.not(
            ElementMatchers.namedOneOf(InflaterInputStream.class.getName())
            .or(ElementMatchers.isPrivate())
                           )
       )

and the AgentBuilder2 this one:

ElementMatchers.namedOneOf(Class.class.getName(), ClassLoader.class.getName());

Now the transformer for AgentBuilder1 uses an Advice to print a log message when the read(...)-method of an InputStream was called, and AgentBuilder2 prints a log message when the ClassLoader.getResource(...)-method or the Class.getResource(...)-method was called.

Everything works as expected, apart from the fact that the AgentBuilder2 logs a lot more resources if AgentBuilder1 was installed than if it wasn't installed - while instrumenting the exact same program... It looks like the Class.getResource(...) method is being called much more often if AgentBuilder1 was installed. Any Ideas?