Each jam has its own sense of etiquette, but these are the points I strive to keep in mind at the Ashland session I co-host.
Traditional music is social. Remember that it's just as much about being out for a coffee or a drink as it is about playing. Consider it a successful day if a lot of fun and laughter was had, even if some incorrect notes were hit.
Keep it a friendly, accessible jam that attracts good players and actively welcomes beginners. It's all about everyone playing and learning together - developing and honing skills and techniques.
We do both Irish tunes (jigs, reels) and old-time fiddle tunes, with the occasional Celtic or bluegrass song. Keep the focus on those styles and don't allow it to drift off into additional kinds of music.
Encourage people to play by ear, but be sympathetic to those who still rely on sheet music. Encourage everyone to participate even on stuff they don't know. Although it's OK to sit out if you like.
Place no restrictions on the number of similar instruments (guitar, bodhran...this is more of an Irish thing), and be liberal with your definition of "traditional" instruments (a plugged in uke bass is fine for example, as is a piano accordion playing old-time). Do emphasize good timing and consistent pace. Taste trumps speed/dexterity.
Continue to introduce new tunes and have others bring their favorite tunes to share. Allow the character of the session to be slightly different each time based on who is there and what they bring to the table.
Participants should leave feeling motivated and empowered to learn more, wanting to improve and wanting to return!