Some filesystem thoughts

[Radio młodych bandytów]

On 24/02/2013 22:45, Ronald Klop wrote:
> On Sun, 24 Feb 2013 11:54:34 +0100, Radio młodych bandytów
> <radiomlodychbandytow at o2.pl> wrote:
>
>> "Ronald Klop" <ronald-freebsd8 at klop.yi.org> wrote:
>>> Creative ideas.
>>> Part of what you want is in fusefs (mounting of files to edit their
>>> content).
>> Mhm. Could you give some link or details in another form?
>
> Just google for 'fusefs'.
> Filesystems based on FUSE:
> http://sourceforge.net/apps/mediawiki/fuse/index.php?title=FileSystems

OK, so you meant zipfs-like things. I thought about file being its own 
mountpoint.

>> And part is implemented in e.g. KDE (integrated support for
>>> various file types in fulltext search and tagging of files/metadata,
>>> etc.).
>> Well, I view it as not much different from implementing a TC / MC /
>> VIM plugin. Anybody can benefit, but they have to implement the right
>> API (And there are several programs that use TC plugins).
>> It's interesting as a way of getting some of these benefits though.
>>
>>> The chances of having all these complex libraries integrated in the
>>> FreeBSD OS are close to zero I presume. But I am not in a position to
>>> decide about that.
>> Frankly, I haven't expected anything different. My thoughts did jump
>> to implementation issues and I see them numerous, but I think the idea
>> itself is not sufficiently mature, so I decided to skip them in the
>> first post.
>>
>>> I think you can't expect the OS to serve everybody's detailed wishes.
>> I don't expect it. I just wanted to discuss an idea that seemed to
>> have a potential.
>>> The OS serves files and user programs know what to do with them.
>> Unfortunately, far too often programs don't know it. Files are often
>> not simple and a single program is unable to deal with them. The only
>> way to deal with such cases ATM that I see is to manually remove
>> layers obfuscating the meaningful sources. In some way, it resembles
>> piping data through multiple programs, except that pipes transport
>> bytes, not files and therefore the transformation has to be performed
>> step by step.
>
> Well. It is probably me, but I don't really get what you're trying to
> say here.
We have
grepmail
mboxgrep
pdfgrep
zgrep
They exist solely because grep doesn't know how to deal with some kinds 
of data. Adding tools doesn't scale as one needs number_of_formats * 
number_of_tools for full coverage. Moving it to another layer would 
reduce it to number_of_formats + number_of_tools.

The approach is not only redundant, but also insufficient because such 
tools don't let you grep f.e. pdfs in gzip email-attachments despite 
having all necessary parts in place.

When we look at the data flow that's necessary to perform such task, it's
unmbox (extracts a list of emails)
  V
unmail (separates individual emails to text and attachments)
  V
ungzip (unzips gzipped attachments)
  V
unpdf (extracts texts)
  V
grep (greps)

If mailboxes contained at most 1 email, emails at most 1 attachment, 
this could be performed as a pipe job.

That's why I say it's similar to piping, yet impossible to implement 
this way.
-- 
Twoje radio