Using Github pages with Jekyll in a subdirectory

I recently wanted to publish my Jekyll site on Github pages, but it seems that putting everything in a subdirectory is giving some issues, even after I change the source to the correct directory.

My structure looks like this:

- site
   - src (contains all Jekyll stuff)
   - README.md
   - GruntFile.js
   - ...

Locally my site builds perfectly and when I go to http://localhost:4040 I can see it just nicely, but when I commit this to my Github and visit username.github.io I get a 404, if I go to username.github.io/src I can see part of my site, however all {% include %} are ignored.

In my _config.yml I updated the source: source: ./src, but that doesn't seem to help.

Is there a way to make Github Pages handle subdirectories properly? Basically I want to tell it that my Jekyll site is inside /src, and I want the url to just be username.github.io instead of username.github.io/src

I know i can use the pages branch and commit to there, but I would prefer if it could happen automaticly.