FreeBSD installers and future direction

Tue May 28 23:56:06 UTC 2013

On May 28, 2013, at 4:16 PM, Super Bisquit wrote:

In the case of firmware loaded systems, all of them aren't going to work with a single boot loader.

Uh…

On the surface, what you're talking about seems to be unrelated to the discussion at-hand.

Nobody said anything about unifying the boot loader.

We're talking about the installer… not the boot loader.

*usually* the installer doesn't care about how it got to its installation environment.

NOTE: I say *usually* because there *are* specialized situations like cobbler + mfsbsd + pc-sysinstall in which case the boot loader needs to provide certain information (which is then accessible through kenv(1)). That type of installation setup will fail if the specialized boot loader does not boot on the desired target installation platform *or* the boot loader that _does_ work on said platform doesn't provide the necessary information.

That being said… bsdinstall doesn't require special knowledge from the boot loader (to the best of my knowledge).

sysinstall didn't need special info from the boot loader either.

I don't see why talking about a unified installer has to include a discussion involving unifying the boot loader (a topic in my mind that is unattainable).
--
Devin

On Tue, May 28, 2013 at 1:15 PM, Teske, Devin <Devin.Teske at fisglobal.com<mailto:Devin.Teske at fisglobal.com>> wrote:

On May 28, 2013, at 8:54 AM, Alfred Perlstein wrote:

On 5/28/13 7:49 AM, Nathan Whitehorn wrote:
On 05/27/13 23:36, Alfred Perlstein wrote:
On 5/27/13 6:53 PM, Nathan Whitehorn wrote:
On 05/27/13 20:40, Alfred Perlstein wrote:
On 5/27/13 2:23 PM, Bruce Cran wrote:
On 27/05/2013 21:28, Alfred Perlstein wrote:
On 5/27/13 11:40 AM, Bruce Cran wrote:
Yes.
Is this a joke?

It probably /was/ too short a reply. Personally I think there should be a single UI and scripting interface across all platforms. We should try and get pc-sysinstall running on all of them first in case there's some problem that means it can't be done, in which case we'd need to use a different backend.

There are just going to be certain platforms that make it EASY to do cool things.  We should embrace that!  That's why there are different platforms!

Some are great for low power, others are great for graphics, cpu power, gpu, networking etc.

If we always go for the lowest common denominator then we are crippling all the platforms for no one's benefit.  Even if something CAN be done, if it is very difficult, or just never happening, then we can't limit everyone's experience based on the more difficult and/or resource strapped platforms.

It's just not good business.

Yes, and all of this cuts both ways: pc-sysinstall has no wireless setup support, for instance. Right now we support what we support because it is the most feature-complete thing we have, not just on tier-2 platforms but also on x86.

To bring this discussion back to the ground, the fact is that we lack an installer that has both internal support for ZFS and a UI. One of the reasons for this is that making a good expressive UI for ZFS is a non-trivial undertaking given its enormous flexibility. The bsdinstall partition editor has been written to be extensible for this, and several people have started writing code to do it, but no one ended up having time to finish. Probably a reasonable thing to do is to start with supporting only a minimal set of features. If anyone felt like actually writing this code, I'm sure it would be appreciated by all and be more productive than email exchanges.
-Nathan

I'm sure if there was a list of reasonable things, such as wireless then pc-sysinstall could be augmented.  This is the first I've heard of that.  All the other complaints have been based on portability.

Is that all that is required now, wireless?

There are more, as well. A partial list of missing features on both sides is here: https://wiki.freebsd.org/PCBSDInstallMerge<https://urldefense.proofpoint.com/v1/url?u=https://wiki.freebsd.org/PCBSDInstallMerge&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=BFNRw%2Fl9%2B28%2BnxLbdzO4ji0pUaEPGB%2BV9%2BaaBuFSGWM%3D%0A&s=3bbe977edd2b122e6d4bc08d7650eb11e412fe03c97b25b3aa86cbf4c96e9ba6>. Other major ones are IPv6 (maybe this has changed?) and no jail setup feature. Most of the existing disk partitioning code in pc-sysinstall, which is the only thing in a FreeBSD installer that is at all complicated, is also *extremely* fragile and needs in all likelihood to be entirely replaced. The merge effort stalled because of this kind of issue -- doing a "merge" rapidly became rewriting both from scratch.
-Nathan

Ah this is so cool.  I'll bring it up with the PCBSD folks today.

Thank you Nathan.

I had my own look at the pc-sysinstall and bsdinstall code and came to the same conclusions, plus some.

One of the biggest obstacles I see is actually a high-level issue that I've self-identified through extensive work on bsdconfig (which is both a back-end and a front-end).

This is the issue of debugging and namespaces.

I've sat down and made lists of other issues… but when I review, I find these issues to be secondary to the above-stated larger issues.

Concretely, I'm saying thus:

+ bsdinstall lacks debugging (debugging is different than logging; from what I could see BSDINSTALL_LOG -- although utilized by both the sh(1) side and the C side -- is only populated during an installation). The ability to have the type of debugging that is in bsdconfig would greatly diminish the amount of time developing important new features.

+ pc-sysinstall lacks debugging (similar situation… producing a log for some action is not the same as being able to have debug statements for the purpose of enhancement the program or troubleshooting an enhancement)

+ bsdinstall separates the backend functionality and the front-end functionality into two separate namespaces (and in the case of C binaries, a third namespace)

+ pc-sysinstall separates one backend into more than one namespace

===

To get an idea of the type of debugging I'm talking about, install sysutils/bsdconfig from the ports tree or install it from a HEAD checkout of base (it's in usr.sbin) and execute:

bsdconfig -d
# produce debugging statements on stdout collated in realtime with the dialog screens

or

bsdconfig -D fooLogfile
# produce debugging statements in "fooLogfile" (debugging statements are hidden from stdout)

or

bsdconfig -D +fooLogfile
# produce debugging statements in both "fooLogfile" *and* on stdout (this is the same as "-dD fooLogfile")

As this stuff gets more modular and there are back-ends, front-ends, global APIs, local APIs, etc. etc.

Having the ability to (after noticing a problem) flip a switch and then get an almost exact location of where you currently-are within the code just by looking at debug statements is extremely helpful in being able to locate the problem.

===

The namespace argument is a bit harder to explain (and quantify for comparison).

In bsdinstall, we see namespace separation most readily by looking at the way the C binaries work.

The namespace separation in this case means that (despite the fact that the C components do a getenv(3) to acquire $BSDINSTALL_LOG, for example) for the large part, the C aspects cannot enjoy code written to extend the sh(1) aspect, and vice-versa.

So if you want a nice debug library that acts in a single way for bsdinstall… that's going to be difficult (but not impossible… you could perhaps cheat by having both the sh(1)-side and the C-side use a logger(1)/syslog(3) facility … but then getting that debug information integrated in the above manner is still non-trivial).

On the other-hand… pc-sysinstall doesn't suffer from the same namespace problems (it's 100% shell), but it's still not conflated as-is done in bsdconfig.

In pc-sysinstall what you'll find is that instead of putting functionality into actual functions… it creates shell scripts that operate in a separate namespace as they are executed as binaries rather than taking advantage of their shell-based nature (in other words… because it's 100% shell, it should perhaps embrace the ability to avoid forking all the time and run everything in a conflated [single] namespace).

===

And as Nathan points out… it quickly starts to look like a rewrite of both to do a merge.

However… the type of "merge" that was being talked about in the above URL (reproduced below from the above reply text for convenience):

https://wiki.freebsd.org/PCBSDInstallMerge<https://urldefense.proofpoint.com/v1/url?u=https://wiki.freebsd.org/PCBSDInstallMerge&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=BFNRw%2Fl9%2B28%2BnxLbdzO4ji0pUaEPGB%2BV9%2BaaBuFSGWM%3D%0A&s=3bbe977edd2b122e6d4bc08d7650eb11e412fe03c97b25b3aa86cbf4c96e9ba6>

Is not a merge that would see a single namespace emerge.

In the above URL, the type of "merge" that's being posited is the kind where bsdinstall becomes a front-end only and will call-out to everything that pc-sysinstall provides.

This could only be bad if pc-sysinstall is left in its current state, because pc-sysinstall expects you to treat the largest portions of its functionality as black-box executables (e.g. you call "delete-part.sh" with arguments… rather than loading a shell library with a delete-part() function).

Debugging is harder when you have to cross a namespace threshold.

Perhaps a better style of merge would be the traditional use of the word…

That is to say, that pc-sysinstall should be slurped *into* bsdinstall (bsdinstall being the newer entity on the block… so it has more up-to-date coding style, albeit not the latest; contrasted to pc-sysinstall's dated inconsistencies within its own code-base -- so a merge in the opposite direction, of giving pc-sysinstall a user interface, would be harder).

However (again, with these however statements)…

If the idea is to merge pc-sysinstall and bsdinstall to solve the issue of too many namespaces and the equally-distression problem of not having a debug layer (which is only marginally helpful if you have a fractured namespace design)…

… I think we could do a lot better.

Perhaps not better with the out-come (which would be hard to judge before the product is truly envisaged), but with respect to disruptiveness.

I recognize that any forward motion in the bsdinstall or pc-sysinstall camp would be disruptive to the people dependent on those technologies.

Meanwhile, nobody is depending on bsdconfig at the moment.

A less (or perhaps, totally non-) disruptive path would be to merge the two entities (bsdinstall and pc-sysinstall) into a third entity (bsdconfig) where the third entity has much more freedom to play.

The end result would be that, when bsdconfig does indeed incorporate all the migrated functionality (and rewritten to achieve the desired enhancements to make development and maintenance easier), we then -- and only then -- disrupt things by wholly replacing them.
--
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
_______________________________________________
freebsd-hackers at freebsd.org<mailto:freebsd-hackers at freebsd.org> mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers<https://urldefense.proofpoint.com/v1/url?u=http://lists.freebsd.org/mailman/listinfo/freebsd-hackers&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=BFNRw%2Fl9%2B28%2BnxLbdzO4ji0pUaEPGB%2BV9%2BaaBuFSGWM%3D%0A&s=f65fc4995680a6eacb77bb87e4ac17599c0146834d5178b300ee6ac88c5038a3>
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org<mailto:freebsd-hackers-unsubscribe at freebsd.org>"

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.