CSS Transition Not Working with Javascript

display is not a property that can be transitioned. You need to make your animation take multiple steps. When you click the button, the modal should be made flex, but it should still be transparent. Then you need to transition the opacity up to 1 which is what CSS transitions can do.

You need to do the inverse whenever you close the modal. Transition back to opacity 0 and after the transition is done, mark it display: none.

var modalOverlay = document.getElementById('modalOverlay');
var modalButton = document.getElementById('modalButton');

modalButton.addEventListener('click', openModal);
window.addEventListener('click', closeModal);

function openModal() {
    // This will cause the browser to know 
    // that the element is display flex for a frame
    requestAnimationFrame(() => {
        modalOverlay.classList.add("modal-overlay--open");
        
        // Then when we wait for the next frame
        // the browser will now know that it needs 
        // to do the transition.  If we don't make
        // them separate actions, the browser
        // will try to optimize the layout and skip
        // the transition
        requestAnimationFrame(() => {
            modalOverlay.classList.add("modal-overlay--open-active");
        });
    });
}

function closeModal(event) {
    if (event.target == modalOverlay) {
        modalOverlay.classList.remove("modal-overlay--open-active");
        setTimeout(() => {
          modalOverlay.classList.remove("modal-overlay--open");
        }, 1100);
    }
}

.modal-overlay {
    display: none;
    opacity: 0;
    position: fixed;
    top: 0;
    left: 0;
    z-index: 10;
    width: 100vw;
    height: 100vh;
    align-items: center;
    justify-content: center;
    background-color: rgba(0,0,0,0.5);
    transition: all 1s ease-in-out;
}

.modal-overlay.modal-overlay--open {
    display: flex;
}

.modal-overlay.modal-overlay--open-active {
    opacity: 1;
}

.modal-box {
    width: 200px;
    height: 200px;
    background-color: blue;
}

<button id="modalButton" class="modal-button">Open Modal</button>
<div id="modalOverlay" class="modal-overlay">
    <div id="modalBox" class="modal-box"></div>
</div>

Let's draw a little insight from how some other frameworks deal with these kinds of transitions. For example, Vue.js separates its enter/leave transitions into 6 sorts of phases:

1. Enter: Starting state, added before entry (for you, display: flex and fully transparent)
2. Enter Active: The transitioning state that sets the transition "target" (opacity towards 1 in your case)
3. Enter To: What it should be after the transition is complete (we aren't going to bother with this)
4. Leave: About to start leaving (nothing really needs to change here for us)
5. Leave Active: Set the target state for your element so that it knows what to transition to (for us, we just remove the class that says opacity: 1)
6. Leave To: We don't need this either

The main thing that we need to consider is that we need the browser to have the element in the page and "rendered" so that it will consider it for transitioning. That's why we add, in our example, the modal-overlay--open class which makes it flex. We then wait just a second and add the transition target class modal-overlay--open-active which causes the element to actually transition.

Then we do the same thing in reverse: remove modal-overlay--open-active so the browser knows to transition the element back to the "normal" style. We set a timeout to remove the display: flex class after the transition is done. You could use event listeners for this, but it's overkill for such an example.