Subject: Re: linux needs bsd networking stack

From: [email-masked] (Larry McVoy)
Date: Wed, 29 May 1996 16:04:37 -0700

:    The "unique" tcp/ip implementation is a liability to linux.
: 
: It could also be one of it's greatest assets, and I think this will
: turn out to be the case.
: 
:    Is anyone working to replace the standard linux stack
:    with port derived from the 4.4BSD code?
: 
: A couple of weeks ago, Larry was babbling to me "oh the stack is
: sloowww, I can't push nearly as much over 100mb/s ether as freebsd
: can, etc."  I said, "thats peculiar" so I did some investigation and
: told Linus about it.  Turned out to be a driver bug and after that was
: fixed the over the wire numbers are unparalleled.

That's not quite true, I think the BSD numbers are still better, I'll
have full lmbench apples to apples runs on the same hardware at the 
end of this week.

: It [bsd] has a well defined architecture, I will agree with lm when he
: mentions that it is a jungle of code to sift through in certain
: respects.  

This is one of my complaints.  The BSD stack has a defined set of "objects"
for dealing with networking; an incomplete list:

	protocol structure for different address families
	interface structure for different media types
	socket structure that cleanly handles different protocols

Another big plus of the BSD stack is tcp_input.c and tcp_output.c.  These
are what most people mean when they say "BSD networking".

Downsides of BSD:

	. I don't particlularly like mbufs; I agree with Linus & Alan that
	they are overkill.  

	. There are layering "invariants" that affect performance: you really
	should allocate your send buffers from the interface driver, because
	it could do some interesting things that would minimize cache flushing.
	I think Van's prototype did this for witless cards.

	. Single processor design.  This is the biggest drawback, IMO.

Proposal/suggestion:

	. Come up with a strawman proposal for the set of "objects" we think
	  we need in Linux.  Do this as part of the work Linus suggested to
	  merge the socket ops with the vfs ops.

	. Steal the TCP code outright.  Nuke the mbuf stuff, use the skbufs
	  or a slightly modified version thereof.

	. Design in SMP support from the start.  This means thinking about
	  thousands of connections running in parallel.

: I think the feeling that the linux stack is "hard to follow" or "has
: very little architecture" has a lot to do with the fact that we don't
: have 20 books analyzing the code c-statement by c-statement like the
: bsd stuff does.  If we had that, I think this desire to use the
: berkeley stack would not be as strong.

Yeah, but a very reasonable point is "we don't have that".  BSD does.
This is a big deal.  Documentation is useful.

---

From: Alan Cox <[email-masked]>
Date: Thu, 30 May 1996 11:06:25 +0100 (BST)

> 	protocol structure for different address families

We have these.

> 	interface structure for different media types

We have these

> 	socket structure that cleanly handles different protocols

We have most of this.

> are what most people mean when they say "BSD networking".

Yep. For 2.0 we have all the core stuff. For pre 2.1 we have stuff going beyond
what BSD is doing - Vegas flow control. If you want to work on stealing and working
stuff in talk with Pedro Roque, get pre 2.1 and take a good look.

> 	I think Van's prototype did this for witless cards.

We could certainly add the allocate via device at very little cost. We just start
using dev->kmalloc(). Note that you can't always win on this a packet may change
route.

> 	. Single processor design.  This is the biggest drawback, IMO.

The Linux one is semi designed for MP work. A given socket can do its own locking,
and there are only a small number of areas of overlap at the moment. Notably

Demultiplex table needs locks
Multiple processor writes in parallel/reads in parallel on datagram requires about
	10 lines of lock code.
We dont run the net_bh() in parallel although most of it we can do very easily.

> 	  merge the socket ops with the vfs ops.

That certainly cant be counter-productive.

> 	  or a slightly modified version thereof.

We can't steal it outright. 4.4BSD has abominable problems as is. The FreeBSD people
have the worst of them fixed but don't have stuff like Vegas and have all the 
horrible spoofing problems caused by bad timers.

> 	  thousands of connections running in parallel.

So long as it still runs very fast for the 99.99% of people with a single CPU 486 PC.
Thats the primary target by market volume.

> This is a big deal.  Documentation is useful.

Its coming. In fact AW should now have released an English language version of the
1.2 kernel programming book. 

Alan