confusion over ports improvements projects intentions (was Re: Is someone already working on a port that supports Boost 1.35.0?)

[Ondrej Fischer]

Hello,

Let me introduce the main ideas, and scope of the ports management 
system, we currently call ports 2.0.

First, the confusion may result from the name, which looks like having 
ambitions to completely replace ports system and infrastructure.
If we call "infrastructure" the ports directory tree with makefiles, 
options, knobs etc. we won't replace/change or even touch this in this 
project. The infrastructure is not the bad in current system, but the good.
If we call management of the ports "the process" (either using make, 
packages or tools like portupgrade), this is what ports 2.0 should solve 
differently, and much more efficiently than any existing tool.

Here we see the main problem in recursive make, which prevents "the 
process" from being parallelized and utilize resources in an optimal 
way. My first draft for improvement was described here: 
http://moon.felk.cvut.cz/~fischeo1/portwizard/

But now we have mouch more powerfull solution. With Aryeh, we designed 
an algorithm, that has a dependency graph of packages on it's input, and 
detects allways the maximum of actions, that can be processed in 
parallel. And the more, it can pass those actions to several processing 
thread pools and thus expolit resources optimally by type.
In other words: If we have 4CPUs and 1GB internet connection, we can use 
eg. 4 parallel builds and eg. 10 parallel fetches allways, when there 
exists a set of such actions, that can be done in parallel.
What is other benefit, is incrementality. You don't have to first choose 
all packages to be installed/upgraded. You just choose a set, and while 
"the process" running, you can add next packages with no harm to 
parallel actions detection. This only needs server -> client model. The 
installation/upgrade process must be one (with several threads).

Right now we are working on prove of concept of underlaying algorithm, a 
prototype, and a detailed specification. The prototype will be in Java, 
since it is extremely easy to implement it, but I guess, it's not 
desired to have java dependency for such system management tool.

That's for introduction.
Regards
Ondrej Fischer


Aryeh M. Friedman wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> David Wood wrote:
> | [Additional CCs added by Aryeh noted, but left untouched]
>
> I removed the OP, Simon and Mezz since this has nothing to boost any 
> more and I changed the Alex Ryba cc to Alejandro Pulver because the 
> first was a misaddressing on my part
> |
> | I wanted to retitle this post, but couldn't come up with a summary 
> of what I was trying to say.
>
> I think the subject now fits better
> |
> |
> | In message <48181A2A.1000303 at gmail.com>, Aryeh M. Friedman 
> <aryeh.friedman at gmail.com> writes
> |> I am top posting because my comments are general and not in 
> relationship to any given point.   I think Mezz along with Simon both 
> the right way to handle it for now.   There are several projects that 
> I am not directly involved designed to tackle this sort of issue with 
> ports 2.0 being the right long term solution but not needed to solve 
> this issue per se so I will not discuss it here.   Most of the 
> projects can be found on Ale's PortsToDo wiki 
> (wiki.freebsd.org/PortsToDo) the main one not found on there is Jim 
> Staplton's virtual ports DB (which I think is either in the committ 
> queue or should be since both me an Ale have signed off on it as being 
> a good idea).  The issue is really not the infrastruct as you state 
> but the more patchs like this we make the more likely it is the 
> infrastruct will become problematic down the road.
> |
> | Forgive me, Aryeh, but you've taken a specific point I made and 
> dismissed it saying that "the infrastructure needs a complete overhaul 
> and I'm the right person to head that work", though you have 
> acknowledged that there is other work going on.
>
> We seem to have a fundimental failure to communicate.   I was saying 
> *SPECFICALLY* that ports 2.0 (at least as currently envisioned) was 
> not the right way to handle this for the time bing (only that if and 
> when ports 2.0 does become adopted that it needs to handle such 
> situations as well or better then the current system).... my original 
> comment was made solely as a user of ports that depend on boost and a 
> desire to make it so there was no python wackiness (which Simon's and 
> Mezz's suggestion does just fine).
>
> As to other people's work odds are that the very core of Ports 2.0 is 
> going to be based on a concept put forward by Ondrej Fischer (I will 
> leave it to him to discuss the details) and Ports 2.0 *ONLY* refers to 
> item 5 in Ale's todo list (which is less then 10% of the total list)
> |
> | I know you will argue you aren't making a reply to my post, so why 
> didn't you trim all my text instead of quoting it? I believe you are 
> replying - and that your reply is pretty close to being non sequitur.
>
> You're the one who attempted to put words into my mouth (totally 
> without cause or instigation) by attempting to make a linkage between 
> my concern about a given current port and my other work in the ports 
> system in general.   The two are completely unrelated.
> |
> | I am not interested in pointless points scoring; I have better 
> things to do. What I want to do is understand the issue..
> |
> |
> | I am no committer - just a mere ports maintainer. Freshports doesn't 
> dig up any ports you maintain, at least not with an email address 
> beginning with aryeh or containing fried.
>
> Since I work closely with Ale who is a committer for book keeping 
> simplicity he is the maintainer of record for several ports we 
> co-maintain: devel/aegis, devel/thistest (besides I wrote the 
> underlaying application), and sysutils/fusefs-ntfs
> |
> | I do not believe that this problem is one that brings into question 
> the whole infrastructure. It certainly isn't "the more patchs (sic) 
> that we make the more likely it is that the infrastruct (sic) will 
> become problematic down the road". Not all evolutionary software 
> engineering is bad, especially if modules are rewritten when 
> appropriate. Of course I accept the possibility of 'bit rot' setting 
> in when code has been through many hands and has many small changes 
> applied. In this case, I believe that things are getting better, not 
> worse or less maintainable.
>
> Neither do I (at least until we break the 25k ports or so [total 
> guess]).  As to evolutionary software development it is the primary 
> model I use and one of the things it teaches is to know when the 
> current design is starting to reach it's limits and change it either 
> by some structural refactoring (if you are lucky) and/or rewrite of 
> major subsystems (if your unlucky)... ports 2.0 takes the point of 
> view that the user level ports system is pretty much good the way it 
> is but the way depends are handled sucks and that this will evenually 
> make the system unworkable unless a completely new depend system is 
> developed and that is why I am always refering to "Recursive Make 
> Considered Harmful"
> |
> | This problem was known about some while back and there's no need to 
> do any fresh engineering. The delay in having a solution available was 
> the need for changes in the base system to support bsd.options.mk. 
> Base system changes mean waiting until all versions of FreeBSD that 
> lack the necessary support are End of Life. After 31 May, ports will 
> be able to process their options and set a dependency on python before 
> including bsd.pre.port.mk.
>
> Please take your head out of "Aryeh is evil" and see my comments for 
> what they where (an attempt to ask if the solution was implemented 
> visa vie boost and boost only!)
> |
> | A simple and elegant solution will soon be available to solve what 
> is essentially a circular dependency at the moment.
>
> It is already solved if you read the reply from simon.
> |
> |
> |
> | I see two issues here, both stemming from the complexities of 
> dependency between ports. Firstly, there's a need to look for the best 
> possible solutions with the tools we have now as well as to look for 
> continuing improvement. Secondly, there's the need to educate both 
> maintainers and administrators on how best to handle the complexities 
> that can occur.
> |
> | I think everyone would agree that the current documentation isn't 
> the best. In stating this, I believe I'm restating Mark Linimon's well 
> known criticism of the current Porters' Handbook. It is not my 
> intention to hurt anyone by what I say. The way out of this is, of 
> course, for people to pay attention to the documentation - but those 
> best placed to do that are some of the busiest people already within 
> the FreeBSD project. Having worked alongside technical authors, I 
> realise what a specialist (and impressive) skill set they have.
>
> This is a non-Ports 2.0 work item already on Ale's todo list 
> spefically to create a more general guide then the porters handbook 
> (his working title is the ports handbook) which is meant for both 
> maintainers and admins.
>
> |
> | At the moment, it is hard for a maintainer to discover current best 
> practice. I suspect this is partly why the same issue is handled in 
> different ways by different ports (as I mentioned above with 
> Kerberos). Meanwhile, it is hard for system administrators to discover 
> how best to use the richness of ports.
>
> I doubt if anyone ever even attempted to compile a best practices list 
> (one Ale's goals with the porters guide).   As to your assertion that 
> all we need is to make maintainers aware of complexities is ass 
> backwards if you ask me when such complexities are symptoms of either 
> bad practices or a bad underlaying system (I personal think it is both)
> |
> |
> | System administrators need the freedom to make whatever decisions 
> they feel are appropriate (as I said, I wouldn't want ports 
> unnecessarily depending on OpenSSL or the Cyrus SASL library). 
> However, features that help administrators manage this richness, such 
> as portconf, are not that well known about, especially by the less 
> technical users.
> |
> |
> |
> | Aryeh - you seem to have something against slave ports. At times, 
> they are very useful. They make the creation and maintenance of 
> client/server ports easy - for example, databases/mysql50-client is a 
> slave port of databases/mysql50-server.
>
> In theory I have nothing against them since they are the best we can 
> do with the current system, *BUT* maintainers all too often 
> inadvertently create problems with them such as the old boost did.
>
> |
> | The reason for both my posts is a feeling that you haven't 
> understood *why* the current situation is as it is, Aryeh. A reply 
> along the lines of "we're working on a complete replacement of the 
> whole infrastructure, and that will solve it" doesn't help with 
> understanding the specific issue that has arisen. You can't offer any 
> guarantee that your new system will solve all the problems - 
> especially if you do not take the time to understand the weaknesses in 
> the current system as well as its strengths.
>
> I think your the one who misunderstood completely by some how assuming 
> I was pushing a non-maintstream agenda (which I agree ports 2.0 
> currently is) instead of simple making a request as user of a specific 
> port.
> |
> | If you bring forward coherent proposals and a proof of concept for 
> Ports 2.0, I will certainly give what input I can at that stage. For 
> now, we live with and continue to improve what we have, rather than 
> looking only to what might replace it if it's found to be better after 
> careful evaluation.
>
> Depend on what Ondrej Fischer feels about discussing his work we can 
> start to put forward such concrete stuff right now (the design is 99% 
> done we are just testing the underlaying alrogithms)
> |
> |
> | Maintainers have to work with OPTIONS as they are. They are a 
> valuable feature, even if they don't offer all the functionality 
> wished for.
>
> Where the (*&(* did you get the impression I was against OPTIONS 
> and/or gnobs?
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (FreeBSD)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkgZHIEACgkQk8GFzCrQm4CZegCdFhZ3TD8vOpPMyzA2BBHpraLJ
> Dw0Anik2dR4BuSZB2OucOzxR/Xiu3t/T
> =3UyF
> -----END PGP SIGNATURE-----
>