Flash disks and FFS layout heuristics

[Martin Fouts]

> -----Original Message-----
> From: Matthew Dillon [mailto:dillon at apollo.backplane.com] 
> Sent: Tuesday, April 01, 2008 3:26 PM
> To: Martin Fouts
> Cc: freebsd-arch at freebsd.org
> Subject: RE: Flash disks and FFS layout heuristics
> 
> 
> What complete bullshit.  If you want to argue technical 
> merits, be my guest.  So far you haven't made one single
> technical point in any of your postings.

My, my. Mr Dillon likes to be rude to people and tell them they are 'stupid' and 'silly', but when he makes naïve comments about systems he doesn't understand and gets called on it, suddenly it's "complete bullshit."

I can see why PHK broke off trying to educate you.

> You've posted about your experience with NAND flash in embedded systems,
> very clearly with SMALL flash devices

Acutally, you're jumping to conclusions, again, Matt.  I mentioned what size devices we used 3 years ago. I haven't spoken at all about the size of devices I have experience with.

> and simple filesystems, and that's fine, it's similar to my 
> flash filesystem

Actually, I mentioned some file systems which are not 'simple' by any reasonable metric.  You're the one who keeps trying to impose 'simple'.

>     experience (which, yes, was primarily on NOR devices but, no, that
>     doesn't magically make you an expert on NAND and me an 
> idiot about it). 

You're not an 'idiot' about NAND, your knowledge is merely limited to reading specs, and as a consequence you're extrapolating beyond that knowledge when you try to apply your theory to NAND, and experience has shown that your extrapolations don't hold up.


> Considering I've pretty much spent my entire life working with hardware
> that is about as ridiculous an assertion as you could make, but clearly
> you believe it.

You need to stop being defensive in technical discussions; stop imposing your presumptions on other peoples problems; and stop thinking that anyone cares enough about you to make any assertions about your background.

I have *not* made any assertions, other than that you've made comments about NAND which betray your lack of experience with it.  You don't have the experience, and your comments about 'trivial' problems, and 'nonexistant' problems clearly shows that.  You don't need to take my word for it. You merely have to check on the state of the art in NAND file systems for CE products.

Oh, you should also stop putting words in my mouth.  You're wrong again.  I've never thought you were an idiot and I don't think so now.  You're rude, arrogant, judgmental, and sure of your of your own skills beyond your actual ability, but you're no idiot.

>     But then you generalized to the entire market and that's not fine.

I've not made any such generalizations, Matt. You're projecting again.  *You* are the one who made the generation that all embedded problems were trivial.

The only thing I speak with authority about in this discussion is convergent CE devices, and then I speak only of the ones I've worked on and what experience with them has been.

> 
> Real filesystems are far more sophisticated then what you 
> will ever see in the embedded flash product,

Now there's a hasty generalization that betrays your attitude problem.  "real" file systems?  First NAND filesystems are 'trivial'.  Then it's 'degenerate'. Now it's not 'real.'

You'll never understand a problem that you dismiss without investigating.

> My interest is squarely with real filesystems targetted
> to mass storage, these days.

Yes. I pointed that out. Also pointed out that as a consequence you're trying to apply approaches that don't work in CE devices.

> 
> I didn't start out smearing people, but if you are going to start
> acting like an asshole then I have no problem ratcheting it up
> to your level.

Dillon, after all these years, I would have thought you'd gotten past that blind spot.  You don't call people 'silly' and 'stupid' and the work they're doing "trivial" and "degenerate" *unless* you're acting like an asshole. As long as I've known you, you've liked starting pissing contests and then blaming the other party.  PHK was wise to have begged off when you started down that path, but I had some time on my hands and thought others would benefit from a technical discussion. If you want a pissing match, I suggest alt.flames, where I'm sure they'll happily accommodate you.

>     Since you don't understand my position, let me lay it out for you
>     in simple terms:
> 
>     * There's no point trying to adapt a flash-unaware filesystem to
>       become flash-aware.  It is a complete waste of time.  

"waste of time" is a value judgment that you don't have the background to make for anyone but yourself. The marketplace, which supports at least two such filesystems, disagrees with your judgment.

> You might as well write a new filesystem.
> If you want to use a flash-unaware
> filesystem you use a translation layer, eat any 
> performance issues, and be done with it.

Congratulations. Welcome to FATFS on usb sticks.

>     * Just because flash-unaware filesystems HAVE To use a 
> translation layer
>       doesn't mean that a translation layer is bad for a flash-aware
>       filesystem.

That is correct.  The FTL approach is suitable for certain types of flash file systems, as I pointed out some number of emails back.  It is not suitable for all.


>     * A named-block translation layer can be an extremely 
> valuable abstraction
>       for use in filesystem designs which directly integrate 
> its features
>       (that is, the filesystem NAMES the block instead of 
> ALLOCATES the
>       block).

'can be' makes for a pretty weak precondition, so sure, it 'can be'.

> 
>       There is absolutely NOTHING inherently bad about the 
> model from a 
>       performance point of view, particularly if your storage 
> media requires
>       relocation (as NAND does).

Either 'relocation' doesn't mean what you think it means, or NAND doesn't require it.

>  The key point is that a 
> named-block layer
>       takes over the functionality of all the indirect 
> pointers that would
>       normally have to be manipulated by higher layers in the 
> filesystem.

Yes. This is what the FTL people do, except the granularity of their named-block is the write unit. It has performance issues.


>       If you can integrate that into the physical storage 
> requirements then
>       you kill two birds with one stone and get major 
> performance benefits
>       from doing so.
> 

That's a big if. It has in practice turned out to be unattainable.  I await your demonstration to the contrary.


>     You are welcome to debate the points, but you'll get 
> burned if you try
>     to take some sort of moral highground stand based on a 
> few piddly flash
>     filesystems written over the course of a few years.  
> Coding at that
>     level is fun and interesting but ultimately not very difficult.

'burned'. 'moral highground'. 'piddly'. 'not very difficult'.  That's a hell of a blindspot to your own behavior that you've got there, Matt.

> Right now my work is with HAMMER.  It's fun to theorize 
> how I could make HAMMER into a flash-aware filesystem but 
> I have no intention of actually doing so any time soon, or ever.
> 

I didn't think so.

> Frankly, if I wanted to write a ground-up flash 
> filesystem I could, it would not be difficult... 

Of course not. People write file systems in undergraduate OS classes.

> But I have no desire to do that at
> this juncture and the lack of desire certainly does not 
> invalidate my comments on the matter.

What 'invalidates' your comments, is that others have tried what you've outlined, in the way that you've outlined it, and it has failed.  That, coupled with your naïve claims about embedded systems not being complex and your several mistaken claims about where the problems are or aren't in such systems simply highlights that, as you say, you're having fun speculating, but, as I say, your speculation would take you down trodden paths to well known conclusions.

> NAND is different from NOR but the differences can be 
> explained pretty much in two paragraphs and most of the same concepts 
> apply.

The interesting aspects lie in the differences.

> It isn't rocket science.

I've done rocket science for a living. It's not that hard, and I've always found that statement silly.

> 
> I am a very technical person.  If you are going to argue 
> merit, then you damn well better say WHY something doesn't
> work, in detail, instead of simply stating that someone
> random other entity couldn't make it work some point in
> the past so therefor it is bad.

You're a 'very technical person' with a very judgmental attitude and a tendency to use emotionally loaded language that you later disclaim.  But no, I don't have to say WHY it doesn't work in detail, provided someone else has already said so. I merely have to point out the existence of the refutation.


> If you do not know the WHY, precisely, then good $#%$#%$#%
> luck designing anything that's actually sophisticated.
> 

"sophisticated", which I suppose is a synonmy for "complex", is an interesting metric for a "very technical person" to apply.

But actually, it's pretty easy to design sophisticated systems when you don't understand the underlying issues. In practice it's more common to make systems more sophisticated in the face of uncertainty, not less.