Is it possible to change firefox's maximize button position in windows?

Yes it is possible to change the order of the three title buttons while the DWM is disabled. But if the DWM is enabled, we are not able to change the order.

Method:

1) copy omni.ja from C:\Program Files (x86)\Mozilla Firefox\browser

2) extract it with your zip tools

3) edit browser.xul under chrome\browser\content\browser, change the order of following content

<hbox id="titlebar-buttonbox-container" align="start">
      <hbox id="titlebar-buttonbox">
        <toolbarbutton class="titlebar-button" id="titlebar-close" command="cmd_closeWindow"/>
        <toolbarbutton class="titlebar-button" id="titlebar-max" oncommand="onTitlebarMaxClick();"/>
        <toolbarbutton class="titlebar-button" id="titlebar-min" oncommand="window.minimize();"/>
      </hbox>

the order of titlebar becomes: close->max->minimized

4) repack the omni.ja with command zip -qr9XD omni.ja * under the omni directory.

5) replace the old omni.ja with the new edited omni.ja

6) the finally result:

Explanation:

1) Firefox's ui layout is based on xul(a xml based layout description language) and browser.xul is used as the description file for the main browser ui.

2) While DWM is enabled, Firefox draws the non-client area using the default window proc(DefWindowProc),

case WM_NCPAINT:
    {
      /*
       * Reset the non-client paint region so that it excludes the
       * non-client areas we paint manually. Then call defwndproc
       * to do the actual painting.
       */

      if (!mCustomNonClient)
        break;

      // let the dwm handle nc painting on glass
      if(nsUXThemeData::CheckForCompositor())
        break;

      HRGN paintRgn = ExcludeNonClientFromPaintRegion((HRGN)wParam);
      LRESULT res = CallWindowProcW(GetPrevWindowProc(), mWnd,
                                    msg, (WPARAM)paintRgn, lParam);
      if (paintRgn != (HRGN)wParam)
        DeleteObject(paintRgn);
      *aRetValue = res;
      result = true;
    }
    break;

So it is impossible to customize the three button while dwm is enabled.