How does the software choose when to start a new page? is it the number of posts or the size of the page? it would seem the size would make more since but it seems that even if the discussion gets long it still doesn't start a new page till there are a certain number of posts. can this be changed?

39 posts to a page. I believe that s because of WebTV limits.

40 actually, remember the original post counts. So the first page will have the original post and 39 replies, and every page following it will have 40 replies.

AFAIK, it's more a matter of server loads. Lots of sites I've seen limit the page sizes to 25 or fewer posts per page, because that way there's less strain being put on the server all at once (or so I presume...) But I think 40 posts per page is the maximum that VB allows for.

i think it'd make more sense to do 100k/page or something to that effect

Mebbe, but that'd require a major fiddle of the forum software, and it'd almost probably send the server load through the roof, as every single post would have to be looked at to determine whether or not it's "fit" on the current page or not. VBulletin doesn't support this feature to my knowledge, and the server's struggling as it is, AFAIK.