How to consume Graph API with a ADAL token?

in my Xamarin.forms project, I use ADAL (Microsoft.IdentityModel.Clients.ActiveDirectory) to authenticate on the Azure portal (Auth 1.0 endpoint). That part work great, but I need to get the security group of the user. So I use this code and passing the token received with ADAL:

 HttpClient client = new HttpClient();
 HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Get, "https://graph.microsoft.com/v1.0/me/memberOf");
 message.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("bearer", token);
 HttpResponseMessage response = await client.SendAsync(message);
 string responseString = await response.Content.ReadAsStringAsync();

I always got StatusCode: 401, ReasonPhrase: 'Unauthorized'. In my azure AD app registration, I add the Graph API and these permissions:

I think I miss something. Any idea?

----- EDIT 1 --- Here my payload. I changed it for a picture for lisibility. I don't know how to post json here

----- EDIT 2 --- OH! I see. I think I need to understand more the Azure Login process. For now I follow an example of ADAL and Azure that let my log and use some function in my backend. So the login process use:

var authContext = new AuthenticationContext(authority); var authResult = await authContext.AcquireTokenAsync(resource, clientId, uri, platformParams);

Where authority = https://login.microsoftonline.com/mysite.com, ResourceID is my backend app ID and clientID is my native app ID. So Shawn is correct, I do not use the Graph.microsoft.com to get the token. Do we have another way to achieve all that? The need of using Graph is only to get the AD group the user has to adjust permission inside the app.