When I transcribe a melody I try I like to determine which of the 7 "church" modes it might be using based on the notes I am hearing. There is a mnemonic for remembering these modes - I Don't Punch Like Muhammed Ali. I learned it in an online music theory class taught by the University of Edinburgh Reid School of Music.
In other words, that's I (Ionian) Don't (Dorian) Punch (Phrygian) Like (Lydian) Muhammad (Mixolydian) A (Aeolian) li (Locrian). Since there are two modes that start with L I try and remember that Locrian comes last. Another one that works is I Don't Play Loud Music Any Longer.
Ionian = the notes from the major scale starting on the 1st scale degree: CDEFGAB.
Dorian = the notes from the major scale starting on the 2nd scale degree: DEFGABC.
Phrygian = the notes from the major scale starting on the 3rd scale degree: EFGABCD.
Lydian = the notes from the major scale starting on the 4th scale degree: FGABCDE.
Mixolydian = the notes from the major scale starting on the 5th scale degree: GABCDEF.
Aeolian = the notes from the major scale starting on the 6th scale degree: ABCDEFG.
Locrian = the notes from the major scale starting on the 7th scale degree: BCDEFGA.
Using the above scenario, if a tune doesn't have any sharps or flats but has G as the tonal center, then it's G-Mixolydian. If if doesn't have any sharps or flats and you feel like D is the tonal center, then it could be D-Dorian.
I used this same methodology when transcribing a melodic portion of the Phish song Horn this week. In the first 30-seconds of the instrumental section that begins after the lyrics are done, I only heard a flattened note once - a Bb - that I treated as an accidental. The rest of the melody was using notes from the C-major scale, although the tonal center of the melody was either G or D. It definitely wasn't C. So I determined for now that the first several bars of of that 2-minute end section of Horn are using either the G-Mixolydian or D-Dorian mode.
More often than not melodies do have lots of sharps (or flats!), so to determine the mode you have think in terms of scale intervals - whole steps, half steps...that kind of thing. As you are transcribing a melody, determine its tonal center. The tonal center is the root note; the note it wants to keep going back to. Once you have the tonal center you can determine the scale/mode based on the other notes being used. For example, a tune using the notes G,A,B,C,D,E,F# with an emphasis on E it would be in E-Aeolian (E-minor).
Once you have the tonal center established, it's really important to pay attention to where the 3rd and 7th scale degrees fall. Is the 3rd major or minor? Is the 7th scale degree flattened? Where those two scale degrees fall plays a big part in determining a tune's mode.