Code First Foreign Keys between two entities

OK what am I missing here or is this just able to be done with data annotation?

I have a Document Entity Model which has a Foreign Key to a User that added the document (one-to-one relationship):

[Table("Documents", Schema = "Configuration")]
public class Document : IPrimaryKey {
    [Key]
    public long Id { get; set; }

    [Required]
    public string OrginalName { get; set; }

    [Required]
    public DocumentTypes DocumentType { get; set; }

    [Required]
    public MIMETypes MIMEType { get; set; }

    [Required]
    public byte[] Data { get; set; }

    [DefaultValue(false)]
    public bool IsPublic { get; set; }

    [Required]
    public DateTimeOffset DateTimeAdded { get; set; }

    [Required]
    public long AddedByUser { get; set; }

    [ForeignKey("AddedByUser")]
    public virtual Details Details { get; set; }
}

I then have a User (Details) Entity that can have an image file (which is stored in the document entities model (none|one-to-one relationship):

[Table("Details", Schema = "User")]
public class Details : IPrimaryKey {
    [Key]
    public long Id { get; set; }

    [Required]
    public string UserId { get; set; }

    [ForeignKey("UserId")]
    public AppUser User { get; set; }

    [Required]
    public string FirstName { get; set; }

    [Required]
    public string LastName { get; set; }

    [CollectionRequired(MinimumCollectionCount = 1)]
    public ICollection<Address> Addresses { get; set; }

    [CollectionRequired(MinimumCollectionCount = 1)]
    public ICollection<Email> Emails { get; set; }

    [CollectionRequired(MinimumCollectionCount = 1)]
    public ICollection<PhoneNumber> PhoneNumbers { get; set; }

    public ICollection<NotificationHistory> NotificationHistory { get; set; }
    public long TimeZoneId { get; set; }

    public long? ImageId { get; set; }

    [ForeignKey("ImageId")]
    public virtual Document Document { get; set; }

    [ForeignKey("TimeZoneId")]
    public virtual TimeZone TimeZone { get; set; }
}

When I try to create a Migration I get this error:

Unable to determine the principal end of an association between the types 'StACS.PeoplesVoice.DataAccessLayer.EntityModels.User.Details' and 'StACS.PeoplesVoice.DataAccessLayer.EntityModels.Configuration.Document'. The principal end of this association must be explicitly configured using either the relationship fluent API or data annotations.

UPDATED:

While still researching this I made two changes and was able to get around the error but this created an unexpected result in my database.

In the Document Entity I added:

public virtual ICollection<Details> Details { get; set; }

In the Details (user) Entity I added:

puflic virtual ICollection<Document> Documents { get; set; }

In my DB Tables I now have the foreign key on the field I want but I have a secondary foreign key for each respectively.

I tried just removing the single virtual reference and left ONLY the ICollection Virtual reference, now I have no foreign key at all.

UPDATED (based on Akash Kava Suggestion):

I have made the following changes [Table("Documents", Schema = "Configuration")] public class Document : IPrimaryKey { [Required] public string OrginalName { get; set; }

    [Required]
    public DocumentTypes DocumentType { get; set; }

    [Required]
    public MIMETypes MIMEType { get; set; }

    [Required]
    public byte[] DocumentData { get; set; }

    [DefaultValue(false)]
    public bool IsPublic { get; set; }

    [Required]
    public DateTimeOffset DateTimeAdded { get; set; }

    [Required]
    public long AddedByUser { get; set; }

    [Key]
    public long Id { get; set; }

    [ForeignKey("AddedByUser")]

    [InverseProperty("Image")]

    public virtual Details User { get; set; }
}

[Table("Details", Schema = "User")]
public class Details : IPrimaryKey {
    [Required]
    public string UserId { get; set; }

    [ForeignKey("UserId")]
    public AppUser User { get; set; }

    [Required]
    public string FirstName { get; set; }

    [Required]
    public string LastName { get; set; }

    [CollectionRequired(MinimumCollectionCount = 1)]
    public ICollection<Address> Addresses { get; set; }

    [CollectionRequired(MinimumCollectionCount = 1)]
    public ICollection<Email> Emails { get; set; }

    [CollectionRequired(MinimumCollectionCount = 1)]
    public ICollection<PhoneNumber> PhoneNumbers { get; set; }

    public ICollection<NotificationHistory> NotificationHistory { get; set; }
    public long TimeZoneId { get; set; }
    public long? ImageId { get; set; }

    [ForeignKey("ImageId")]
    [InverseProperty("User")]
    public Document Image { get; set; } 

    [ForeignKey("TimeZoneId")]
    public virtual TimeZone TimeZone { get; set; }


    [Key]
    public long Id { get; set; }
}

I have commented out the Fluent API Code

Unable to determine the principal end of an association between the types 'StACS.PeoplesVoice.DataAccessLayer.EntityModels.User.Details' and 'StACS.PeoplesVoice.DataAccessLayer.EntityModels.Configuration.Document'. The principal end of this association must be explicitly configured using either the relationship fluent API or data annotations.