Consider the following situation, you have some text within TinyMCE that you select. You then choose the list option, however instead of just the few "lines" converting to a list, the entire contets of the editor becomes a single list item. What's gone wrong?
The problem is very common to everyone who chooses to use <P> tags and instead relies upon multiple <BR> to create the illusion of Paragraphs. You will mostly likely also encounter problems with alignment and indentation.
Under normal circumstances, if you have three lines, each enclosed in a <P> tag, then the list simply replaces the <P> with an <LI>. When you don't have this, then it looks for the nearest block element, in this case the entire document, and replaces it with an <LI> resulting in a single list item containing the entire document.
A colleague of mine wrote a series of articles on P tags (The Importance and Practicalities of P Tags) and covers some of the issues people claim as reasons to not use P tags. It's worth reading to help gain a better understanding of how to use P tags, but meet your end users visual expectations.