Open source business models fall into one of the following categories and often companies have hybrids of these:
1). Dual license. Company, releases their software under open source and another commercial license. Often the commercial license has some sought after features. In reality, most often there really isn't much of a difference other than the commercial version is easier to install and has support bundled with it.
2). Packaging. Company packages up the software, puts it in a pretty box (either shrink-wrap or tin-wrapped), and sells supporting manuals and documentation.
3). Support. The most obvious. Company provides support for the software. Who are you going to go to as a legit company who is using the software? Jim Bob or the guys who wrote it and maintain it.
4). Consulting. Company sells integration, business process re-engineering, whatever, as value adds to the software. It's really about provide a solution rather than just some bits.
5). Leech. This isn't really open source, but instead is a company making money off an open source project/company. Example: there exists some cool open source application, let's say a database. Company A comes along and writes some proprietary application, let's say middleware that allows one to easily move from a proprietary db to an open source one with little to no change in the code. Company A saves their customers 10s of thousands of dollars and rakes in the cash by leeching off the open source db.
As for barriers to entry, well...are you talking about to starting a project (none) or usurping the business model of an existing companies project? If it's the latter I defer to the previous comment: Who are you going to go to? Jim Bob or the guys who wrote and mange the project? Historically it's been the latter. Also, Jim Bob has a learning curve the others don't (most often).
How many years for a project to reach the tipping point? There is no exact formula for this. It depends on the project, the team driving it, public interest, etc...lots of factor. Perhaps game theory could provide some meaningful estimations. I'm not sure.
