poudriere, Go and networking

[Steve Wills]

Hi,

On 12/11/2015 10:55, Malcolm Matalka wrote:
> Piotr Florczyk <piotr.florczyk at gemius.com> writes:
> 
>> W dniu 11.12.2015 o 15:36, Kurt Jaeger pisze:
>>> Hi!
>>>
>>>> Recently I had to package couple of programs written in Go and godep is
>>>> becoming the standard for dependency tracking in Go projects.
>>>> For example I currently had to package telegraf. Here is the thing. Poudriere
>>>> disables networking after fetch phase and I don't know before extract
>>>> phase what dependencies are inside.
>>>
>>> We recently upgraded maven, the java-world 'make and godep' and all
>>> the ports that need maven to build have the same problem, see:
>>>
>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188110#c37
>>>

I'll just note here that Go at least downloads source, where I believe
Maven downloads pre-compiled jars (correct me if I'm wrong), so
comparing them isn't completely apples to apples.

Perhaps in the maven case having a port for each dep does make sense.

At the moment, I see quite a few ports with pre-compiled jars in their
distinfo.

>>>> So here is the question: would it be possible to have networking
>>>> enabled during extract phase ?
>>>> Or maybe there is another solution (some flag in ports maybe that
>>>> I'm missing ?)

I don't think packaging each go dep separately is a good idea, though
I've noticed the Fedora project does it.

Some ports of go apps do list each dep as a separate dist file and this
seems the best way to me. But I can understand it's too much work for
some ports and it does increase the work a great deal.

Have you considered using "godep restore" and making your own dist file?

This has the advantage of being quick, easy, reliable and arguably
correct as it is just doing what upstream expects a developer to do.
It's still better than just packaging up the compiled binary that the
upstream distributes and preserves the local build and lets users
inspect source easily if they want. The downside is wasted disk space
for distfiles, but it's not a huge amount as go projects aren't
typically very large source bases.

>>>
>>> I think we need some fancy fetch target per distfile which basically
>>> uses technology-dependend (maven, godep, etc) ways to trigger
>>> the 'fetch' during the fetch-phase. Probably some sort
>>> of base-fetch vrs. dep-fetch ?
>>>
>> New target might not be needed but I think this is good idea. Altough it does not solve my problem with poudriere. In my case, the soonest I
>> can fetch dependencies is in post-extract target. So if poudriere didn't cut off networking at this stage we wouldn't need any changes and
>> every one would be happy.
> 
> This sounds like it would be a security hole to let a package download
> extra things that the FreeBSD package system does not know about and
> cannot validate.
> 

Yes, this is not something we want in general. Using "godep restore" and
making your own distfile at least prevents the deps from changing later.
Though "godep restore" should be using the git checksums and comparing
those, I don't think "go get" does. So "godep restore and making your
distfile is more secure.

>> Even if we come up with proper solution it will require cutting off network at some later stage than post-extract. In my opinion we might
>> aswell move it to that point right now.
> 
> Perhaps you should make a tool which takes a go project as input and a
> FreeBSD package as output?
> 

This is an interesting idea. It's a bit of work though and you're be
re-implementing pkg in go and chasing any changes it made. And I'm not
sure how it would integrate with ports.

Steve