Why this wrong IIFE function returns this

tl;dr: The function doesn't return anything since it isn't executed.

A JavaScript interpreter is going to parse this as two statements:

A function declaration

function A() {}

followed by an expression statement

("hello",6)

The expression statement consists of the grouping operator, containing the comma operator which returns the result of the right operand, which is 6.

As mentioned in the comments, if you want this to be interpreted as an IIFE, wrap the function definition (or the whole line) in parenthesis:

(function A() {})("hello",6)
// or
(function A() {}("hello",6))