From bdrewery at FreeBSD.org Sat Feb 1 14:40:40 2014 From: bdrewery at FreeBSD.org (Bryan Drewery) Date: Sat, 01 Feb 2014 08:40:33 -0600 Subject: [patch] TLS Server Name Indication (SNI) support for fetch(1) In-Reply-To: <52BECBE8.8080906@gibfest.dk> References: <20130608205653.GA8765@ogoshi.int.nbs-system.com> <52BECBE8.8080906@gibfest.dk> Message-ID: <52ED0761.5000301@FreeBSD.org> On 12/28/2013 7:02 AM, Thomas Steen Rasmussen wrote: > On 08-06-2013 22:56, Sofian Brabez wrote: >> Hi, >> >> fetch(1) currently does not support TLS extension Server Name >> Indication (RFC >> 6066) [1] when dealing with SSL. Nowadays lot of clients and servers >> implement >> this extension. > Hello! > > fetch(1) is still missing SNI support as of r259440 - any chance of > seeing this patch committed ? > As ipv4 depletion gets worse we will see SSL websites using SNI more and > more. This is overdue. > > Thanks, and may you all have a wonderful new year! > > /Thomas Steen Rasmussen This was added in head r258347 Nov 19 2013: http://svnweb.freebsd.org/changeset/base/258347 It made it to stable/10 before 10.0 and into stable/9. It works if you install ca_root_nss cert.pem: > # pkg install ca_root_nss > ... > # ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem > ... > # fetch -v -o - https://sni.velox.ch|head -n 15 > looking up sni.velox.ch > connecting to sni.velox.ch:443 > SSL options: 81004bff > Peer verification enabled > Using CA cert file: /etc/ssl/cert.pem > Verify hostname > SSL connection established using ECDHE-RSA-AES256-GCM-SHA384 > Certificate subject: /C=CH/ST=Zuerich/L=Zuerich/O=Kaspar Brand/CN=*.sni.velox.ch > Certificate issuer: /C=BM/O=QuoVadis Limited/OU=www.quovadisglobal.com/CN=QuoVadis Global SSL ICA > requesting https://sni.velox.ch/ > - > > > 5063 BTLS SNI Test Site: *.sni.velox.ch > > 945 kBps > 00m00s

TLS SNI Test Site: *.sni.velox.ch

> > >

Great! Your client [fetch libfetch/2.0] > sent the following TLS server name indication extension > (RFC 6066) > in its ClientHello (negotiated protocol: TLSv1.2, cipher suite: ECDHE-RSA-AES256-GCM-SHA384):

> 
  sni.velox.ch
>

In your request, this header was included:

> 
  Host: sni.velox.ch
I'm not sure what the plan is for a base CA file, but adding ca_root_nss does allow it to work. -- Regards, Bryan Drewery -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 553 bytes Desc: OpenPGP digital signature URL: From yuri at rawbw.com Sat Feb 1 22:55:36 2014 From: yuri at rawbw.com (Yuri) Date: Sat, 01 Feb 2014 14:55:35 -0800 Subject: r_debug in ld-elf.so.1 should be a public symbol to allow for an alternative run-time loader Message-ID: <52ED7B67.2090801@rawbw.com> I think it would be reasonable if r_debug and related symbols in the ld-elf.so.1 run-time loader would be made public. Currently, only r_debug_symbol is public out of all debugger-related symbols in ld-elf.so. This is done so that debuggers can add a breakpoint on it to get notified of changes in the set of loaded shared libraries. This is sufficient when ld-elf.so is the only one who loads objects. Now imagine the situation when an application wants to load its own ELF objects in its own way, still having the benefit of using the debug info. Such application would need to have access to the list of objects maintained by ld-elf.so. r_debug variable contains this list. Such application would necessarily be system specific, and would have to have the knowledge of some inner workings of ld-elf.so. But, in my opinion, this is a legitimate use and should be allowed. Therefore I am suggesting this patch. Except r_debug, it adds related lock object and functions to the list of publics, because they are needed to avoid race conditions. This is, of course, not a mainstream way to do things, but nevertheless it should be allowed. This is also a minor patch, and it can't possibly hurt anything else. --begin patch-- --- libexec/rtld-elf/Symbol.map (revision 260894) +++ libexec/rtld-elf/Symbol.map (working copy) @@ -15,6 +15,11 @@ dlinfo; dl_iterate_phdr; r_debug_state; + r_debug; + rtld_bind_lock; + rlock_acquire; + wlock_acquire; + lock_release; __tls_get_addr; }; --end patch-- Yuri From rmirzazadeh at gmail.com Sun Feb 2 10:07:59 2014 From: rmirzazadeh at gmail.com (Reza Mirzazadeh) Date: Sun, 2 Feb 2014 13:37:37 +0330 Subject: remove shutdown messages in console Message-ID: hi . how can i remove shutdown messages in console ? or even print my arbitrary messages in console ? *** FINAL System shutdown message from root at geekiam *** System going down IMMEDIATELY some messages like these ! i don't want it print theses messages -- From jhellenthal at dataix.net Sun Feb 2 11:02:20 2014 From: jhellenthal at dataix.net (Jason Hellenthal) Date: Sun, 2 Feb 2014 06:02:14 -0500 Subject: remove shutdown messages in console In-Reply-To: References: Message-ID: <2D6DF212-BF65-4784-B466-8F595DF2059B@dataix.net> Either configure syslogd correctly or use sysctl consmute -- Jason Hellenthal Voice: 95.30.17.6/616 JJH48-ARIN > On Feb 2, 2014, at 5:07, Reza Mirzazadeh wrote: > > hi . how can i remove shutdown messages in console ? or even print my > arbitrary messages in console ? > > > *** FINAL System shutdown message from root at geekiam *** > System going down IMMEDIATELY > > some messages like these ! i don't want it print theses messages > -- > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6118 bytes Desc: not available URL: From killing at multiplay.co.uk Mon Feb 3 09:12:18 2014 From: killing at multiplay.co.uk (Steven Hartland) Date: Mon, 3 Feb 2014 09:12:10 -0000 Subject: iconv support in 10.x compatibility with closed source apps? Message-ID: <00018D706D2B4278A74A58BE482BF968@multiplay.co.uk> Since 10.x includes iconv in base the converts/libiconv port refuses to build with the error: ===> libiconv-1.14_1 converters/libiconv should not be used with iconv from base. Please fix the port which tries to use it. Now thats all well and good if this was port but its not its a piece of close source software which is linked to libiconv.so.3 So without the option of recompiling the application with base iconv what's the correct fix? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster at multiplay.co.uk. From tijl at FreeBSD.org Mon Feb 3 10:29:23 2014 From: tijl at FreeBSD.org (Tijl Coosemans) Date: Mon, 3 Feb 2014 11:29:13 +0100 Subject: iconv support in 10.x compatibility with closed source apps? In-Reply-To: <00018D706D2B4278A74A58BE482BF968@multiplay.co.uk> References: <00018D706D2B4278A74A58BE482BF968@multiplay.co.uk> Message-ID: <20140203112913.47bfc11b@kalimero.tijl.coosemans.org> On Mon, 3 Feb 2014 09:12:10 -0000 Steven Hartland wrote: > Since 10.x includes iconv in base the converts/libiconv port > refuses to build with the error: > > ===> libiconv-1.14_1 converters/libiconv should not be used > with iconv from base. Please fix the port which tries to use it. > > Now thats all well and good if this was port but its not its > a piece of close source software which is linked to libiconv.so.3 > > So without the option of recompiling the application with base > iconv what's the correct fix? Update your ports tree and install converters/libiconv. The ban has been lifted. From rysto32 at gmail.com Mon Feb 3 16:41:14 2014 From: rysto32 at gmail.com (Ryan Stone) Date: Mon, 3 Feb 2014 11:41:13 -0500 Subject: Races in ichsmb(9) when accessed from a multithreaded process Message-ID: ichsmb.c has the following rather worrisome comment: * This driver assumes that the generic SMBus code will ensure that * at most one process at a time calls into the SMBus methods below. However, when I look at the code is sys/dev/smbus, I see nothing that actually guarantees this if two threads in the same process call ioctls on the same file descriptor. It does call smbus_request_bus, but mostly that just calls down into the smbus implementation (in this case ichsmb) with SMBUS_CALLBACK. ichsmb always just acks the request, so no actual locking ends up occurring. Is it intended that smb(9) clients be required to do their own locking? It seems to me that that is way more fragile than it needs to be. From w-info2 at xs4all.nl Mon Feb 3 19:42:03 2014 From: w-info2 at xs4all.nl (Photo stuff) Date: Mon, 03 Feb 2014 20:33:11 +0100 Subject: The sonewconn listen queue overflow issue Message-ID: <52EFEEF7.5010704@xs4all.nl> L.S. I came across a lot of messages in the log since, I think, about 9.1, that sometimes fill it up so much that all else is made invisible, of the type: sonewconn: pcb 0xyyyyyyyyyyyyyyyy: Listen queue overflow: 8 already in queue awaiting acceptance I searched a bit on the web and came across recommendations to try netstat -nAa to find out which program this came from. Well, running netstat -nAa |grep pcb 0xyyyyyyyyyyyyyyyy in a loop didn't work, it didn't give any output even though the messages kept coming in the log during that time. I forgot about it then recently searched a bit more, and I found this page: http://lawrencechen.net/2014/sonewconn-pcb-0xfffffe006acd9310-listen-queue So a listen queue overflow of 8 is caused by a listen value of 5 (QLEN > 3 * (QLIM / 2)) Doing: netstat -LaAn | grep '/5 ' resulted in ffffzzzzzzzzzzzz tcp4 0/0/5 127.0.0.1:8000 The ffffzzz value was not the value in the message log, but the connection means in my case it had to be the junkbuster proxy (which I still use, still works well :) ), as I didn't run anything else locally. So I looked into junkbuster's cource and in bind.c I changed 2 instances of: while (listen(fd, 5) == -1) { to while (listen(fd, 10) == -1) { /* 10 instead of 5, this fixes dmesg spamming of the type 'sonewconn: pcb 0xyyyyyyyyyyyyyyyy: Listen queue overflow: 8 already in queue awaiting acceptance'? */ This improved the situation, but still gave the issue of the log filling up too much. So then I changed it to 20, which gave me silence :) Checking the latest log I couldn't find anything in the last 10 days. So for people who have this issue I recommend calculating which value the listen-value that the overflow-value corresponds to, then checking as above and then it should be possible to find the daemon causing the issue. And modify that program... But while this removes the errors, what do these messages really signify? I mean which didn't this happen before in earlier versions of FreeBSD? Some people have recommended just to stop using certain programs but that doesn't seem a solution. listen(2) doesn't give a real clue. It mentions changes to the queue in 4.5, so very long ago. Haven't read commit logs to see if there are hints there :) Btw., at the moment (running 10.0 RC2) my message log now gets filled up with something else :) hwpstate0: set freq failed, err 6 This happened long ago already in 9.1 with my AMD3500+ and still now on my AMD FX6100... Regards, Wouter From ian at FreeBSD.org Mon Feb 3 20:28:47 2014 From: ian at FreeBSD.org (Ian Lepore) Date: Mon, 03 Feb 2014 13:28:43 -0700 Subject: Races in ichsmb(9) when accessed from a multithreaded process In-Reply-To: References: Message-ID: <1391459323.13026.100.camel@revolution.hippie.lan> On Mon, 2014-02-03 at 11:41 -0500, Ryan Stone wrote: > ichsmb.c has the following rather worrisome comment: > > * This driver assumes that the generic SMBus code will ensure that > * at most one process at a time calls into the SMBus methods below. > > However, when I look at the code is sys/dev/smbus, I see nothing that > actually guarantees this if two threads in the same process call > ioctls on the same file descriptor. It does call smbus_request_bus, > but mostly that just calls down into the smbus implementation (in this > case ichsmb) with SMBUS_CALLBACK. ichsmb always just acks the > request, so no actual locking ends up occurring. > > Is it intended that smb(9) clients be required to do their own > locking? It seems to me that that is way more fragile than it needs > to be. It looks to me like most of the SMBUS_CALLBACK implementations are stubbed out, and thus are wrong. The notable exceptions are iicbus/iicsmb.c which relies on the underlying iic driver to get it right (and I know at least some of them do), and bktr/bktr_i2c.c which has what looks like good exclusive-bus-grant logic (modulo the use of Giant for locking). -- Ian From torek at elf.torek.net Mon Feb 3 21:20:51 2014 From: torek at elf.torek.net (Chris Torek) Date: Mon, 3 Feb 2014 14:01:00 -0700 (MST) Subject: Races in ichsmb(9) when accessed from a multithreaded process In-Reply-To: Message-ID: <201402032101.s13L10rm092760@elf.torek.net> Yes, we found that the code does not properly allow either multiple threads *or* multiple processes to work. The top level driver makes it a single-open device (which has the obvious drawback that only one process can open() the device). But even then, multiple threads within a process, or forking, or using PF_LOCAL sockets to hand over the file descriptor, result in races. We have a hack that makes it work for our case but the SMB framework needs an overhaul. Chris From julian at freebsd.org Tue Feb 4 03:27:08 2014 From: julian at freebsd.org (Julian Elischer) Date: Tue, 04 Feb 2014 11:26:52 +0800 Subject: remove shutdown messages in console In-Reply-To: References: Message-ID: <52F05DFC.7070007@freebsd.org> On 2/2/14, 6:07 PM, Reza Mirzazadeh wrote: > hi . how can i remove shutdown messages in console ? or even print my > arbitrary messages in console ? > > > *** FINAL System shutdown message from root at geekiam *** > System going down IMMEDIATELY > > some messages like these ! i don't want it print theses messages > -- > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" > we implimented the consmute sysctl option to "mute" an otherwise configured console for exactly this purpose. (and boot messages). From feld at FreeBSD.org Tue Feb 4 18:30:51 2014 From: feld at FreeBSD.org (Mark Felder) Date: Tue, 04 Feb 2014 12:30:49 -0600 Subject: opteron a1100 arm In-Reply-To: References: Message-ID: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> On Fri, Jan 31, 2014, at 12:13, Wojciech Puchar wrote: > AMD just presented it's ARM based CPU and motherboard. is support in > FreeBSD planned and what is it's status? > Is it even available to purchase? From riggs at freebsd.org Tue Feb 4 21:00:49 2014 From: riggs at freebsd.org (Thomas Zander) Date: Tue, 4 Feb 2014 22:00:47 +0100 Subject: opteron a1100 arm In-Reply-To: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> Message-ID: On 4 Feb 2014 19:31, "Mark Felder" wrote: > > On Fri, Jan 31, 2014, at 12:13, Wojciech Puchar wrote: > > AMD just presented it's ARM based CPU and motherboard. is support in > > FreeBSD planned and what is it's status? > > > > Is it even available to purchase? No but it may well be an early reminder of the upcoming generation of powerful ARM servers that we don't want to leave unsupported. Riggs From jim at netgate.com Tue Feb 4 21:21:06 2014 From: jim at netgate.com (Jim Thompson) Date: Tue, 4 Feb 2014 15:21:03 -0600 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> Message-ID: <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> On Feb 4, 2014, at 3:00 PM, Thomas Zander wrote: > On 4 Feb 2014 19:31, "Mark Felder" wrote: >> >> On Fri, Jan 31, 2014, at 12:13, Wojciech Puchar wrote: >>> AMD just presented it's ARM based CPU and motherboard. is support in >>> FreeBSD planned and what is it's status? >>> >> >> Is it even available to purchase? > > No but it may well be an early reminder of the upcoming generation of > powerful ARM servers that we don't want to leave unsupported. ?may well be? isn?t that attractive when the 8-core, 64-bit Intel C2000 parts are here, now, at a lower TDP (20W, .vs 25W for the a1100. 22nm rocks). When Intel moves the C2K series to 14nm later this year, the power savings will close the door. The opteron a1100 samples in March, I wouldn?t expect boards before Summer. Jim From jim at netgate.com Tue Feb 4 21:42:29 2014 From: jim at netgate.com (Jim Thompson) Date: Tue, 4 Feb 2014 15:42:19 -0600 Subject: opteron a1100 arm In-Reply-To: <60555.1391549390@critter.freebsd.dk> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> Message-ID: <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> On Feb 4, 2014, at 3:29 PM, Poul-Henning Kamp wrote: > In message <493DEB39-C4B4-409E-B8B2-B1B11E013754 at netgate.com>, Jim Thompson wri > tes: > >>> No but it may well be an early reminder of the upcoming generation of >>> powerful ARM servers that we don't want to leave unsupported. >> >> isn't that attractive when the 8-core, 64-bit Intel C20 >> 00 parts are here, now, at a lower TDP >> (20W, .vs 25W for the a1100. 22nm rocks). > > I very much welcome a competing 64bit CPU into the marketplace and > will buy one myself, as soon as I can, for no other reason than to > help break the X86 monopoly on server architecture. > > Monopolies are never a good thing. True, but I didn?t say that the chip wasn?t interesting. What I said is that it?s not that attractive (to the real market for these: micro servers). The dual 10Gig Ethernet and 8 SATA 3.0 ports are interesting. You won?t get that with a C2K system at 25W TDP, (4 x GigE that can run at 2.5Gbps per port, and 2 SATA 3.0 ports currently) but Intel owns IP for both, so if that becomes a differentiator for design wins, I?d expect a future variant to cover. But by all means, port FreeBSD to it. Perhaps it can be the long-desired ?reference platform? to bring ARM into a ?Tier 1? architecture status. Jim From phk at phk.freebsd.dk Tue Feb 4 21:36:59 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 04 Feb 2014 21:29:50 +0000 Subject: opteron a1100 arm In-Reply-To: <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> Message-ID: <60555.1391549390@critter.freebsd.dk> In message <493DEB39-C4B4-409E-B8B2-B1B11E013754 at netgate.com>, Jim Thompson wri tes: >> No but it may well be an early reminder of the upcoming generation of >> powerful ARM servers that we don't want to leave unsupported. > >isn't that attractive when the 8-core, 64-bit Intel C20 >00 parts are here, now, at a lower TDP >(20W, .vs 25W for the a1100. 22nm rocks). I very much welcome a competing 64bit CPU into the marketplace and will buy one myself, as soon as I can, for no other reason than to help break the X86 monopoly on server architecture. Monopolies are never a good thing. ... Not even if they're FOSS software: LLVM did wonders for the GCC projects drive and attitude. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ian at FreeBSD.org Tue Feb 4 22:10:00 2014 From: ian at FreeBSD.org (Ian Lepore) Date: Tue, 04 Feb 2014 15:09:55 -0700 Subject: opteron a1100 arm In-Reply-To: <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: <1391551795.1196.11.camel@revolution.hippie.lan> On Tue, 2014-02-04 at 15:42 -0600, Jim Thompson wrote: > On Feb 4, 2014, at 3:29 PM, Poul-Henning Kamp wrote: > > > In message <493DEB39-C4B4-409E-B8B2-B1B11E013754 at netgate.com>, Jim Thompson wri > > tes: > > > >>> No but it may well be an early reminder of the upcoming generation of > >>> powerful ARM servers that we don't want to leave unsupported. > >> > >> isn't that attractive when the 8-core, 64-bit Intel C20 > >> 00 parts are here, now, at a lower TDP > >> (20W, .vs 25W for the a1100. 22nm rocks). > > > > I very much welcome a competing 64bit CPU into the marketplace and > > will buy one myself, as soon as I can, for no other reason than to > > help break the X86 monopoly on server architecture. > > > > Monopolies are never a good thing. > > True, but I didn?t say that the chip wasn?t interesting. What I said is that it?s not that attractive (to the real market for these: micro servers). > > The dual 10Gig Ethernet and 8 SATA 3.0 ports are interesting. You won?t get that with a C2K system at 25W TDP, (4 x GigE that can run at 2.5Gbps per port, and 2 SATA 3.0 ports currently) but Intel owns IP for both, so if that becomes a differentiator for design wins, I?d expect a future variant to cover. > > But by all means, port FreeBSD to it. Perhaps it can be the long-desired ?reference platform? to bring ARM into a ?Tier 1? architecture status. > > Jim We have no shortage of ARM platforms. IMO the thing that prevents ARM from becoming tier 1 is manpower. -- Ian From lev at FreeBSD.org Tue Feb 4 22:12:08 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Wed, 5 Feb 2014 02:12:01 +0400 Subject: opteron a1100 arm In-Reply-To: <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: <28896996.20140205021201@serebryakov.spb.ru> Hello, Jim. You wrote 5 ??????? 2014 ?., 1:42:19: JT> But by all means, port FreeBSD to it. Perhaps it can be the JT> long-desired ?reference platform? to bring ARM into a ?Tier 1? architecture status. It is ARMv8A, which is differs from ARMv5/6/7 as i385 from amd64. So, we should talks about TWO architectures here, as we talk about i386 and amd64. -- // Black Lion AKA Lev Serebryakov From lev at FreeBSD.org Tue Feb 4 22:15:42 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Wed, 5 Feb 2014 02:15:36 +0400 Subject: opteron a1100 arm In-Reply-To: <28896996.20140205021201@serebryakov.spb.ru> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> <28896996.20140205021201@serebryakov.spb.ru> Message-ID: <1801933421.20140205021536@serebryakov.spb.ru> Hello, Lev. You wrote 5 ??????? 2014 ?., 2:12:01: JT>> But by all means, port FreeBSD to it. Perhaps it can be the JT>> long-desired ?reference platform? to bring ARM into a ?Tier 1? architecture status. LS> It is ARMv8A, which is differs from ARMv5/6/7 as i385 from amd64. So, we LS> should talks about TWO architectures here, as we talk about i386 and amd64. I'm not even sure, that we have compiler for ARMv8, as we have very old gcc, and is LLVM AArch64 backend production-ready? -- // Black Lion AKA Lev Serebryakov From phk at phk.freebsd.dk Tue Feb 4 21:48:21 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 04 Feb 2014 21:48:20 +0000 Subject: opteron a1100 arm In-Reply-To: <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: <60681.1391550500@critter.freebsd.dk> In message <23B18B88-D888-46B3-99F6-905F86E20FAF at netgate.com>, Jim Thompson wri tes: >The dual 10Gig Ethernet and 8 SATA 3.0 ports are interesting. You >won't get that with a C2K system at 25W TDP [...] Couple that with some decent SSD disks and you have a system that should be able to pump out close to 400k HTTP requests a second running Varnish... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From feld at FreeBSD.org Tue Feb 4 22:31:28 2014 From: feld at FreeBSD.org (Mark Felder) Date: Tue, 04 Feb 2014 16:31:27 -0600 Subject: opteron a1100 arm In-Reply-To: <60681.1391550500@critter.freebsd.dk> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> <60681.1391550500@critter.freebsd.dk> Message-ID: <1391553087.27234.79361889.59679194@webmail.messagingengine.com> On Tue, Feb 4, 2014, at 15:48, Poul-Henning Kamp wrote: > In message <23B18B88-D888-46B3-99F6-905F86E20FAF at netgate.com>, Jim > Thompson wri > tes: > > >The dual 10Gig Ethernet and 8 SATA 3.0 ports are interesting. You > >won't get that with a C2K system at 25W TDP [...] > > Couple that with some decent SSD disks and you have a system that > should be able to pump out close to 400k HTTP requests a second > running Varnish... > Wow that would be amazing From bapt at FreeBSD.org Tue Feb 4 22:35:34 2014 From: bapt at FreeBSD.org (Baptiste Daroussin) Date: Tue, 4 Feb 2014 23:35:28 +0100 Subject: opteron a1100 arm In-Reply-To: <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> Message-ID: <20140204223528.GF23872@ithaqua.etoilebsd.net> On Tue, Feb 04, 2014 at 03:21:03PM -0600, Jim Thompson wrote: > > On Feb 4, 2014, at 3:00 PM, Thomas Zander wrote: > > > On 4 Feb 2014 19:31, "Mark Felder" wrote: > >> > >> On Fri, Jan 31, 2014, at 12:13, Wojciech Puchar wrote: > >>> AMD just presented it's ARM based CPU and motherboard. is support in > >>> FreeBSD planned and what is it's status? > >>> > >> > >> Is it even available to purchase? > > > > No but it may well be an early reminder of the upcoming generation of > > powerful ARM servers that we don't want to leave unsupported. > > ?may well be? isn?t that attractive when the 8-core, 64-bit Intel C2000 parts are here, now, at a lower TDP > (20W, .vs 25W for the a1100. 22nm rocks). > > When Intel moves the C2K series to 14nm later this year, the power savings will close the door. > > The opteron a1100 samples in March, I wouldn?t expect boards before Summer. > > Jim > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" A board like this will allow portmgr to provide official and regular package built for arm! so yes for sure we do need to support that! regards, Bapt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From cbergstrom at pathscale.com Tue Feb 4 22:40:54 2014 From: cbergstrom at pathscale.com (=?UTF-8?B?IkMuIEJlcmdzdHLDtm0i?=) Date: Wed, 05 Feb 2014 05:37:24 +0700 Subject: opteron a1100 arm In-Reply-To: <20140204223528.GF23872@ithaqua.etoilebsd.net> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> Message-ID: <52F16BA4.4060202@pathscale.com> On 02/ 5/14 05:35 AM, Baptiste Daroussin wrote: > On Tue, Feb 04, 2014 at 03:21:03PM -0600, Jim Thompson wrote: >> On Feb 4, 2014, at 3:00 PM, Thomas Zander wrote: >> >>> On 4 Feb 2014 19:31, "Mark Felder" wrote: >>>> On Fri, Jan 31, 2014, at 12:13, Wojciech Puchar wrote: >>>>> AMD just presented it's ARM based CPU and motherboard. is support in >>>>> FreeBSD planned and what is it's status? >>>>> >>>> Is it even available to purchase? >>> No but it may well be an early reminder of the upcoming generation of >>> powerful ARM servers that we don't want to leave unsupported. >> ?may well be? isn?t that attractive when the 8-core, 64-bit Intel C2000 parts are here, now, at a lower TDP >> (20W, .vs 25W for the a1100. 22nm rocks). >> >> When Intel moves the C2K series to 14nm later this year, the power savings will close the door. >> >> The opteron a1100 samples in March, I wouldn?t expect boards before Summer. I don't track the Atom family and not sure if C2000 is the new out-of-order version, but until that is available - this is no comparison by a long shot. Also 5w on the cpu package could easily be made up for in a superior motherboard and chassis design. Think about how much power interconnects takes, PCIe and everything else.. From phk at freebsd.org Tue Feb 4 22:49:28 2014 From: phk at freebsd.org (Poul-Henning Kamp) Date: Tue, 04 Feb 2014 22:49:24 +0000 Subject: opteron a1100 arm In-Reply-To: <52F16BA4.4060202@pathscale.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> Message-ID: <60985.1391554164@critter.freebsd.dk> In message <52F16BA4.4060202 at pathscale.com>, =?UTF-8?B?IkMuIEJlcmdzdHLDtm0i?= w rites: > Think about how much power interconnects takes, PCIe and everything else.. Not to mention the raw savings of avoiding all the crap that has been nailed inexpertly onto the X86 platform over the years... Did you notice that "X86 arduino" Intel just launched ? Does anybody but me cringe at the thought of an Arduino with UEFI, ACPI, microcode updates and SMM ticking away in the background to make sure it doesn't run too hot ? If ARM and AMD are smart, an ARM64 machine will have the OS loaded and handed over to in a matter of seconds rather than minutes, and that alone would make a lot of people love them. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From jim at netgate.com Tue Feb 4 23:31:22 2014 From: jim at netgate.com (Jim Thompson) Date: Tue, 4 Feb 2014 17:31:12 -0600 Subject: opteron a1100 arm In-Reply-To: <1391553087.27234.79361889.59679194@webmail.messagingengine.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> <60681.1391550500@critter.freebsd.dk> <1391553087.27234.79361889.59679194@webmail.messagingengine.com> Message-ID: <7DAAAE33-2502-410D-A840-5EFB6EB8EE0B@netgate.com> On Feb 4, 2014, at 4:31 PM, Mark Felder wrote: > > > On Tue, Feb 4, 2014, at 15:48, Poul-Henning Kamp wrote: >> In message <23B18B88-D888-46B3-99F6-905F86E20FAF at netgate.com>, Jim >> Thompson wri >> tes: >> >>> The dual 10Gig Ethernet and 8 SATA 3.0 ports are interesting. You >>> won't get that with a C2K system at 25W TDP [...] >> >> Couple that with some decent SSD disks and you have a system that >> should be able to pump out close to 400k HTTP requests a second >> running Varnish... >> > > Wow that would be amazing and then there?s Sandstorm http://conferences.sigcomm.org/hotnets/2013/papers/hotnets-final43.pdf From nwhitehorn at freebsd.org Tue Feb 4 23:37:43 2014 From: nwhitehorn at freebsd.org (Nathan Whitehorn) Date: Tue, 04 Feb 2014 17:37:42 -0600 Subject: opteron a1100 arm In-Reply-To: <60985.1391554164@critter.freebsd.dk> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> Message-ID: <52F179C6.2050400@freebsd.org> On 02/04/14 16:49, Poul-Henning Kamp wrote: > In message <52F16BA4.4060202 at pathscale.com>, =?UTF-8?B?IkMuIEJlcmdzdHLDtm0i?= w > rites: > >> Think about how much power interconnects takes, PCIe and everything else.. > Not to mention the raw savings of avoiding all the crap that has > been nailed inexpertly onto the X86 platform over the years... > > Did you notice that "X86 arduino" Intel just launched ? > > Does anybody but me cringe at the thought of an Arduino with UEFI, ACPI, > microcode updates and SMM ticking away in the background to make sure > it doesn't run too hot ? > This AMD platform has UEFI and ACPI, so that's at least 2 out of 4... -Nathan From jim at netgate.com Wed Feb 5 00:23:17 2014 From: jim at netgate.com (Jim Thompson) Date: Tue, 4 Feb 2014 18:23:08 -0600 Subject: opteron a1100 arm In-Reply-To: <52F16BA4.4060202@pathscale.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> Message-ID: <8537A333-176E-4DF4-A712-2ADFE1B75338@netgate.com> On Feb 4, 2014, at 4:37 PM, C. Bergstr?m wrote: > I don't track the Atom family and not sure if C2000 is the new out-of-order version, but until that is available - this is no comparison by a long shot. Yes, the C2000 SoCs and Bay Trail are Silvermont architecture. Silvermont has OOE and a better branch predictor than did Bonnell or Saltwell. At 22nm, Intel had enough die area to just add in more cores rather than rely on ?hyper threading" for better threaded performance so Hyper Threading isn?t part of this SoC. The ISA is roughly Westmere (think 2010 Core architecture). AES-NI, SSE4.1/4.2 and extended page tables (so: bhyve) are all ?in there?. > Also 5w on the cpu package could easily be made up for in a superior motherboard and chassis design. Think about how much power interconnects takes, PCIe and everything else.. That?s why the infrastructure guys ( ? Dell, HP, NEC, Ericsson, Quanta, Supermicro, etc) are building micro server systems with 450 ?nodes? per rack. From jim at netgate.com Wed Feb 5 00:35:25 2014 From: jim at netgate.com (Jim Thompson) Date: Tue, 4 Feb 2014 18:35:15 -0600 Subject: opteron a1100 arm In-Reply-To: <52F179C6.2050400@freebsd.org> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> <52F179C6.2050400@freebsd.org> Message-ID: On Feb 4, 2014, at 5:37 PM, Nathan Whitehorn wrote: > On 02/04/14 16:49, Poul-Henning Kamp wrote: >> In message <52F16BA4.4060202 at pathscale.com>, =?UTF-8?B?IkMuIEJlcmdzdHLDtm0i?= w >> rites: >> >>> Think about how much power interconnects takes, PCIe and everything else.. >> Not to mention the raw savings of avoiding all the crap that has >> been nailed inexpertly onto the X86 platform over the years... >> >> Did you notice that "X86 arduino" Intel just launched ? >> >> Does anybody but me cringe at the thought of an Arduino with UEFI, ACPI, >> microcode updates and SMM ticking away in the background to make sure >> it doesn't run too hot ? >> > > This AMD platform has UEFI and ACPI, so that's at least 2 out of 4... > -Nathan The ?X86 arduino? (?Gallileo?) has a Quark 1100 SoC running at 400MHz. It implements a 32-bit Pentium instruction set and has a TDP of 1.9W - 2.2W. I don?t think it?s going to ?run too hot?, and yes it runs SMM, UEFI and ACPI-compatible sleep states. From adrian at freebsd.org Wed Feb 5 02:09:47 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Tue, 4 Feb 2014 18:09:46 -0800 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> <52F179C6.2050400@freebsd.org> Message-ID: On 4 February 2014 16:35, Jim Thompson wrote: > > On Feb 4, 2014, at 5:37 PM, Nathan Whitehorn wrote: > >> On 02/04/14 16:49, Poul-Henning Kamp wrote: >>> In message <52F16BA4.4060202 at pathscale.com>, =?UTF-8?B?IkMuIEJlcmdzdHLDtm0i?= w >>> rites: >>> >>>> Think about how much power interconnects takes, PCIe and everything else.. >>> Not to mention the raw savings of avoiding all the crap that has >>> been nailed inexpertly onto the X86 platform over the years... >>> >>> Did you notice that "X86 arduino" Intel just launched ? >>> >>> Does anybody but me cringe at the thought of an Arduino with UEFI, ACPI, >>> microcode updates and SMM ticking away in the background to make sure >>> it doesn't run too hot ? >>> >> >> This AMD platform has UEFI and ACPI, so that's at least 2 out of 4... >> -Nathan > > The "X86 arduino" ("Gallileo") has a Quark 1100 SoC running at 400MHz. It implements a 32-bit Pentium instruction set and has a TDP of 1.9W - 2.2W. > > I don't think it's going to "run too hot", and yes it runs SMM, UEFI and ACPI-compatible sleep states. We'd still have to port to that platform. I'd give it go if someone would ship me one. :) -a From adrian at freebsd.org Wed Feb 5 02:37:56 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Tue, 4 Feb 2014 18:37:54 -0800 Subject: opteron a1100 arm In-Reply-To: <52F19E8F.1030101@pathscale.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> <52F179C6.2050400@freebsd.org> <52F19E8F.1030101@pathscale.com> Message-ID: The first project is "make 32 bit uefi booting work." That's kind of a requirement. -a From adrian at freebsd.org Wed Feb 5 02:13:23 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Tue, 4 Feb 2014 18:13:21 -0800 Subject: opteron a1100 arm In-Reply-To: <60985.1391554164@critter.freebsd.dk> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> Message-ID: On 4 February 2014 14:49, Poul-Henning Kamp wrote: > In message <52F16BA4.4060202 at pathscale.com>, =?UTF-8?B?IkMuIEJlcmdzdHLDtm0i?= w > rites: > >> Think about how much power interconnects takes, PCIe and everything else.. > > Not to mention the raw savings of avoiding all the crap that has > been nailed inexpertly onto the X86 platform over the years... > > Did you notice that "X86 arduino" Intel just launched ? > > Does anybody but me cringe at the thought of an Arduino with UEFI, ACPI, > microcode updates and SMM ticking away in the background to make sure > it doesn't run too hot ? > > If ARM and AMD are smart, an ARM64 machine will have the OS loaded > and handed over to in a matter of seconds rather than minutes, and > that alone would make a lot of people love them. My experience with ARM and power management is that as the chips get more recent, the voodoo hiding behind black boxes increases. Qualcomm wouldn't even contemplate open sourcing the power management stuff. I have more faith that the intel stuff will at least be sanely usable by us OS hackers over what the multiplude of ARM vendors have given us. It looks like a mess and I've only vaguely peripherally looked at it. -a From cbergstrom at pathscale.com Wed Feb 5 02:18:12 2014 From: cbergstrom at pathscale.com (=?ISO-8859-1?Q?=22C=2E_Bergstr=F6m=22?=) Date: Wed, 05 Feb 2014 09:14:39 +0700 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> <52F179C6.2050400@freebsd.org> Message-ID: <52F19E8F.1030101@pathscale.com> On 02/ 5/14 09:09 AM, Adrian Chadd wrote: > On 4 February 2014 16:35, Jim Thompson wrote: >> On Feb 4, 2014, at 5:37 PM, Nathan Whitehorn wrote: >> >>> On 02/04/14 16:49, Poul-Henning Kamp wrote: >>>> In message <52F16BA4.4060202 at pathscale.com>, =?UTF-8?B?IkMuIEJlcmdzdHLDtm0i?= w >>>> rites: >>>> >>>>> Think about how much power interconnects takes, PCIe and everything else.. >>>> Not to mention the raw savings of avoiding all the crap that has >>>> been nailed inexpertly onto the X86 platform over the years... >>>> >>>> Did you notice that "X86 arduino" Intel just launched ? >>>> >>>> Does anybody but me cringe at the thought of an Arduino with UEFI, ACPI, >>>> microcode updates and SMM ticking away in the background to make sure >>>> it doesn't run too hot ? >>>> >>> This AMD platform has UEFI and ACPI, so that's at least 2 out of 4... >>> -Nathan >> The "X86 arduino" ("Gallileo") has a Quark 1100 SoC running at 400MHz. It implements a 32-bit Pentium instruction set and has a TDP of 1.9W - 2.2W. >> >> I don't think it's going to "run too hot", and yes it runs SMM, UEFI and ACPI-compatible sleep states. > We'd still have to port to that platform. > > I'd give it go if someone would ship me one. :) If a few of these could be procured - I wonder if this is a good GSOC topic? Anyone know any students who might be interested? From superbisquit at gmail.com Wed Feb 5 05:35:42 2014 From: superbisquit at gmail.com (Joe Nosay) Date: Wed, 5 Feb 2014 00:35:40 -0500 Subject: WebRTC Message-ID: Is it stupid for me to try to port WebRTC plus dependencies to FreeBSD? I am not good at programming; but, I try to find and/or ask for solutions so that a program can work in/on FreeBSD. What I am aware of is that even if it is a good idea, unless it has been presented by someone more well known or higher up in the food chain, it will be ignored. I could say," Hey, know what? If we start on such-and-such right now, then FreeBSD will be at the same level as X and Y in this." However, I know that such words are useless from someone on the bottom. As I states, my knowledge is limited; yet, a starting effort has been done for webrtc. I can't do much more. Why? Because I am homeless and sick, that's why. Adrian Chadd asked on Google+ about FreeBSD having webrtc; and, I decided to try to start it. I'm stuck at the beginning step of porting a dependency. I re-edited the file paths in /usr/local/include/unicode/$ITEM.c_OR_h . I decided to present this idea to the list even after being taunted. There is no reason to worry about who says what. From lev at FreeBSD.org Wed Feb 5 09:54:05 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Wed, 5 Feb 2014 13:53:52 +0400 Subject: opteron a1100 arm In-Reply-To: <20140204223528.GF23872@ithaqua.etoilebsd.net> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> Message-ID: <18209443.20140205135352@serebryakov.spb.ru> Hello, Baptiste. You wrote 5 ??????? 2014 ?., 2:35:28: BD> A board like this will allow portmgr to provide official and regular package BD> built for arm! so yes for sure we do need to support that! Again: it is ARMv8. So, board like this will allow portmgr to provide official and regular package built for arm in same sense, that amd64-based board will help to build packages for i386 :) -- // Black Lion AKA Lev Serebryakov From bapt at FreeBSD.org Wed Feb 5 10:05:00 2014 From: bapt at FreeBSD.org (Baptiste Daroussin) Date: Wed, 5 Feb 2014 11:04:54 +0100 Subject: opteron a1100 arm In-Reply-To: <18209443.20140205135352@serebryakov.spb.ru> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <18209443.20140205135352@serebryakov.spb.ru> Message-ID: <20140205100453.GH23872@ithaqua.etoilebsd.net> On Wed, Feb 05, 2014 at 01:53:52PM +0400, Lev Serebryakov wrote: > Hello, Baptiste. > You wrote 5 ??????? 2014 ?., 2:35:28: > > BD> A board like this will allow portmgr to provide official and regular package > BD> built for arm! so yes for sure we do need to support that! > Again: it is ARMv8. So, board like this will allow portmgr to provide official and regular package > built for arm in same sense, that amd64-based board will help to build > packages for i386 :) > Yes and we do build i386 packages on and amd64 box :) regards, Bapt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From phk at phk.freebsd.dk Wed Feb 5 08:01:12 2014 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 05 Feb 2014 08:01:02 +0000 Subject: opteron a1100 arm In-Reply-To: <52F179C6.2050400@freebsd.org> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> <52F179C6.2050400@freebsd.org> Message-ID: <62389.1391587262@critter.freebsd.dk> In message <52F179C6.2050400 at freebsd.org>, Nathan Whitehorn writes: >This AMD platform has UEFI and ACPI, so that's at least 2 out of 4... Ahh crap! -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From kevin.bowling at kev009.com Wed Feb 5 08:05:16 2014 From: kevin.bowling at kev009.com (Kevin Bowling) Date: Wed, 05 Feb 2014 00:55:22 -0700 Subject: opteron a1100 arm In-Reply-To: <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: On 2/4/2014 2:42 PM, Jim Thompson wrote: > > On Feb 4, 2014, at 3:29 PM, Poul-Henning Kamp wrote: > >> In message <493DEB39-C4B4-409E-B8B2-B1B11E013754 at netgate.com>, Jim Thompson wri >> tes: >> >>>> No but it may well be an early reminder of the upcoming generation of >>>> powerful ARM servers that we don't want to leave unsupported. >>> >>> isn't that attractive when the 8-core, 64-bit Intel C20 >>> 00 parts are here, now, at a lower TDP >>> (20W, .vs 25W for the a1100. 22nm rocks). >> >> I very much welcome a competing 64bit CPU into the marketplace and >> will buy one myself, as soon as I can, for no other reason than to >> help break the X86 monopoly on server architecture. >> >> Monopolies are never a good thing. > > True, but I didn?t say that the chip wasn?t interesting. What I said is that it?s not that attractive (to the real market for these: micro servers). > > The dual 10Gig Ethernet and 8 SATA 3.0 ports are interesting. You won?t get that with a C2K system at 25W TDP, (4 x GigE that can run at 2.5Gbps per port, and 2 SATA 3.0 ports currently) but Intel owns IP for both, so if that becomes a differentiator for design wins, I?d expect a future variant to cover. > > But by all means, port FreeBSD to it. Perhaps it can be the long-desired ?reference platform? to bring ARM into a ?Tier 1? architecture status. > > Jim Where A1100 wins hands down is memory capacity, and possibly even memory bandwidth (DDR4 mentioned in the PR..). Intel's server Atom chips and even the extremely powerful Xeon E3 are quite limited by 32GB RAM (and it's also somewhat expensive vs RDIMMs). This just doesn't cut it for a ton of workloads either family of Intel CPU is otherwise well suited for. The A1100 is going to force Intel to fix that nonsense quickly or see a rapid exodus on the low end. From jim at netgate.com Wed Feb 5 13:45:00 2014 From: jim at netgate.com (Jim Thompson) Date: Wed, 5 Feb 2014 07:44:55 -0600 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: > On Feb 5, 2014, at 1:55, Kevin Bowling wrote: > [...] > Where A1100 wins hands down is memory capacity, and possibly even memory bandwidth (DDR4 mentioned in the PR..). > > Intel's server Atom chips and even the extremely powerful Xeon E3 are quite limited by 32GB RAM (and it's also somewhat expensive vs RDIMMs). I'm not sure what your point is. The C2750, C2550, C2558 & C2758 will all address 64 GB of DDR3 RAM. You'll have to use Registered DIMMs and likely drop back to DDR3 to get to 128GB on the A1100. So I see parity in terms of addressable RAM. From george+freebsd at m5p.com Wed Feb 5 14:09:38 2014 From: george+freebsd at m5p.com (George Mitchell) Date: Wed, 05 Feb 2014 09:09:31 -0500 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: <52F2461B.1070405@m5p.com> On 02/05/14 08:44, Jim Thompson wrote: > > >> On Feb 5, 2014, at 1:55, Kevin Bowling wrote: >> > [...] > >> Where A1100 wins hands down is memory capacity, and possibly even memory bandwidth (DDR4 mentioned in the PR..). >> >> Intel's server Atom chips and even the extremely powerful Xeon E3 are quite limited by 32GB RAM (and it's also somewhat expensive vs RDIMMs). > > I'm not sure what your point is. > > The C2750, C2550, C2558 & C2758 will all address 64 GB of DDR3 RAM. > > You'll have to use Registered DIMMs and likely drop back to DDR3 to get to 128GB on the A1100. > > So I see parity in terms of addressable RAM. > > [...] This discussion is very entertaining. But let's not lose sight of the main point: ARM processors are not going away. It is to FreeBSD's detriment if we don't have Tier 1 support for a processor that, by some guesses, already comprises the majority of new installations by count. -- George From Kamil.Choudhury at anserinae.net Wed Feb 5 14:34:55 2014 From: Kamil.Choudhury at anserinae.net (Kamil Choudhury) Date: Wed, 5 Feb 2014 14:34:50 +0000 Subject: opteron a1100 arm In-Reply-To: <52F2461B.1070405@m5p.com> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> , <52F2461B.1070405@m5p.com> Message-ID: George wrote: > This discussion is very entertaining. But let's not lose sight of the > main point: ARM processors are not going away. It is to FreeBSD's > detriment if we don't have Tier 1 support for a processor that, by some > guesses, already comprises the majority of new installations by count. Moving on to specifics, what concrete steps would be necessary to actually get FreeBSD support for this architecture? _______________________________________________ freebsd-hackers at freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" From george+freebsd at m5p.com Wed Feb 5 15:02:55 2014 From: george+freebsd at m5p.com (George Mitchell) Date: Wed, 05 Feb 2014 10:02:48 -0500 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> , <52F2461B.1070405@m5p.com> Message-ID: <52F25298.9000307@m5p.com> On 02/05/14 09:34, Kamil Choudhury wrote: > > George wrote: >> This discussion is very entertaining. But let's not lose sight of the >> main point: ARM processors are not going away. It is to FreeBSD's >> detriment if we don't have Tier 1 support for a processor that, by some >> guesses, already comprises the majority of new installations by count. > > Moving on to specifics, what concrete steps would be necessary to actually > get FreeBSD support for this architecture? [...] https://wiki.freebsd.org/ARMTier1 -- George From lidl at pix.net Wed Feb 5 17:23:17 2014 From: lidl at pix.net (Kurt Lidl) Date: Wed, 05 Feb 2014 12:23:14 -0500 Subject: opteron a1100 arm In-Reply-To: <62389.1391587262@critter.freebsd.dk> References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <20140204223528.GF23872@ithaqua.etoilebsd.net> <52F16BA4.4060202@pathscale.com> <60985.1391554164@critter.freebsd.dk> <52F179C6.2050400@freebsd.org> <62389.1391587262@critter.freebsd.dk> Message-ID: <52F27382.1030505@pix.net> On 2/5/14 3:01 AM, Poul-Henning Kamp wrote: > In message <52F179C6.2050400 at freebsd.org>, Nathan Whitehorn writes: > >> This AMD platform has UEFI and ACPI, so that's at least 2 out of 4... > > Ahh crap! > It's actually more than that. The AMD64 platform has been called out for a much greater deal of standardization than the multitude of different options in the various existing ARM SoC boards. The phrase you want to search for is: ARM Server Base System Architecture Of course, the reference doc: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0029/index.html Is only available to registered ARM customers. But, my understanding is that UEFI is the only boot option, ACPI is the power management system, the USB2 controllers must support EHCI 1.1, the USB3 controllers must support XHCI 1.0 and all SATA controllers must support AHCI 1.3. They're attempting to drive the platform to a more standardized set of components and boot methods (and hypervisor support) so that if your software works on vendor A's SBSA board, it should also work on vendor B's SBSA board. -Kurt From kevin.bowling at kev009.com Wed Feb 5 18:11:51 2014 From: kevin.bowling at kev009.com (Kevin Bowling) Date: Wed, 05 Feb 2014 11:11:30 -0700 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: On 2/5/2014 6:44 AM, Jim Thompson wrote: > > >> On Feb 5, 2014, at 1:55, Kevin Bowling wrote: >> > [...] > >> Where A1100 wins hands down is memory capacity, and possibly even memory bandwidth (DDR4 mentioned in the PR..). >> >> Intel's server Atom chips and even the extremely powerful Xeon E3 are quite limited by 32GB RAM (and it's also somewhat expensive vs RDIMMs). > > I'm not sure what your point is. Simply that this is a better chip than Intel's current low end offerings because all relevant bandwidths are much higher (RAM, I/O, net) and these matter more to many DC workloads. > The C2750, C2550, C2558 & C2758 will all address 64 GB of DDR3 RAM. > > You'll have to use Registered DIMMs and likely drop back to DDR3 to get to 128GB on the A1100. > > So I see parity in terms of addressable RAM. That's half, and a big deal if you want to run multiple VMs or a large ARC or many large JVM heaps or any number of other things. You're trying to claim the chip is already matched by the current Atom, where it is clearly not, so I'll fire it back at you - I'm not sure what your point is. I'm happy to see the announcement and am looking forward to running FreeBSD on the platform with others. From dieterbsd at gmail.com Wed Feb 5 18:41:26 2014 From: dieterbsd at gmail.com (Dieter BSD) Date: Wed, 5 Feb 2014 10:41:24 -0800 Subject: opteron a1100 arm Message-ID: Jim writes: > So I see parity in terms of addressable RAM. Parity is better than nothing (which seems to be what most machines provide these days :-( ), but ECC is much better. :-) Speaking of new machines and ECC, anyone heard any news about an ETA for Berlin? (and mainboards for it that support ECC?) (google finds nothing) From adrian at freebsd.org Wed Feb 5 19:35:41 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Wed, 5 Feb 2014 11:35:39 -0800 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> Message-ID: On 5 February 2014 10:11, Kevin Bowling wrote: > On 2/5/2014 6:44 AM, Jim Thompson wrote: >> >> >> >>> On Feb 5, 2014, at 1:55, Kevin Bowling wrote: >>> >> [...] >> >>> Where A1100 wins hands down is memory capacity, and possibly even memory >>> bandwidth (DDR4 mentioned in the PR..). >>> >>> Intel's server Atom chips and even the extremely powerful Xeon E3 are >>> quite limited by 32GB RAM (and it's also somewhat expensive vs RDIMMs). >> >> >> I'm not sure what your point is. > > > Simply that this is a better chip than Intel's current low end offerings > because all relevant bandwidths are much higher (RAM, I/O, net) and these > matter more to many DC workloads. > > >> The C2750, C2550, C2558 & C2758 will all address 64 GB of DDR3 RAM. >> >> You'll have to use Registered DIMMs and likely drop back to DDR3 to get to >> 128GB on the A1100. >> >> So I see parity in terms of addressable RAM. > > > That's half, and a big deal if you want to run multiple VMs or a large ARC > or many large JVM heaps or any number of other things. You're trying to > claim the chip is already matched by the current Atom, where it is clearly > not, so I'll fire it back at you - I'm not sure what your point is. > > I'm happy to see the announcement and am looking forward to running FreeBSD > on the platform with others. But that's not enough - some one / some company has to step up and do a port, then run with it long enough in -HEAD to make it stable for general use. Almost all of the ARM stuff in FreeBSD doesn't fall under this category. It's getting better, but in a lot of instances the ports are done for very specific chipsets and/or use cases and that doesn't stress the rest of the system out enough. The MIPS port went through this too, FWIW - it initially was stable if you never bothered doing much in userland, but userland suffered from a lot of random hilarity. so if you were a router that occasionally ran ntpd, it was great. If you were an AP that kept going into hostapd to do things, it would get upset. "make all install clean" in a port directory wasn't great. It was only through repeated head-slamming from Warner, Juli and others that this stuff got ironed out enough. So, I'm all for the platform - who's going to port it? Step up please and be cheered. :-P -a From imp at bsdimp.com Thu Feb 6 20:57:06 2014 From: imp at bsdimp.com (Warner Losh) Date: Thu, 6 Feb 2014 13:56:54 -0700 Subject: opteron a1100 arm In-Reply-To: References: <1391538649.19169.79261269.3C5F49D1@webmail.messagingengine.com> <493DEB39-C4B4-409E-B8B2-B1B11E013754@netgate.com> <60555.1391549390@critter.freebsd.dk> <23B18B88-D888-46B3-99F6-905F86E20FAF@netgate.com> , <52F2461B.1070405@m5p.com> Message-ID: On Feb 5, 2014, at 7:34 AM, Kamil Choudhury wrote: > > George wrote: >> This discussion is very entertaining. But let's not lose sight of the >> main point: ARM processors are not going away. It is to FreeBSD's >> detriment if we don't have Tier 1 support for a processor that, by some >> guesses, already comprises the majority of new installations by count. > > Moving on to specifics, what concrete steps would be necessary to actually > get FreeBSD support for this architecture? You'll need a new pmap for arm, and a bunch of other things. Plus you'll need to get ACPI working on arm64, and you may also need FDT + ACPI coexistence depending on your development route. Plus you'll need arm64 (aarch64) support in llvm and/or some external toolchain (which would mean you'd need to beef up the external toolchain support for freebsd's build). And then you'll need new drivers. And people to test it and hammer it and fix the stability issues that pop up. And that's not even considering the issues with pkg support etc. It is a heavy lift to get all the way there, although a 'hack' port that gets to single user with a ram-disk or nfs root and a serial port working wouldn't be too hard. So it's like Ian said: we need bodies working on this... So far, I'm only aware of one part timer. Warner From imp at bsdimp.com Thu Feb 6 21:00:54 2014 From: imp at bsdimp.com (Warner Losh) Date: Thu, 6 Feb 2014 14:00:51 -0700 Subject: remove shutdown messages in console In-Reply-To: <52F05DFC.7070007@freebsd.org> References: <52F05DFC.7070007@freebsd.org> Message-ID: <1D565840-1ADA-4567-85B0-FB4AE771B823@bsdimp.com> On Feb 3, 2014, at 8:26 PM, Julian Elischer wrote: > On 2/2/14, 6:07 PM, Reza Mirzazadeh wrote: >> hi . how can i remove shutdown messages in console ? or even print my >> arbitrary messages in console ? >> >> >> *** FINAL System shutdown message from root at geekiam *** >> System going down IMMEDIATELY >> >> some messages like these ! i don't want it print theses messages >> -- >> _______________________________________________ >> freebsd-hackers at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" >> > we implimented the consmute sysctl option to "mute" an otherwise configured console for exactly this purpose. > (and boot messages). Except these are different beast. They aren't kernel messages, but generated by userland... to get rid of these, hack /etc/syslog.conf to get rid of the *.emerg * line. Or hack shutdown to stop sending them.... Or use 'halt' and 'reboot' only Warner From lev at FreeBSD.org Fri Feb 7 13:50:07 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Fri, 7 Feb 2014 17:49:57 +0400 Subject: Why clang bundled with 10-RELEASE doesn't define __has_feature(cxx_nullptr)? Message-ID: <717736061.20140207174957@serebryakov.spb.ru> Hello, freebsd-stable. Why __has_feature(cxx_nulptr) doesn't work on system compiler? nullptr itself works! Here is sample program: ============================= test.cxx #include #ifndef __has_feature #warning "Doesn't support for __has_feature at all" #define __has_feature(x) 0 #endif #if __has_feature(cxx_nullptr) #warning "Has nullptr" #else #warning "Doesn't have nullptr" #endif int main(int argn, char *argv[]) { void *v = nullptr; return 0; } ====================================== > c++ --version FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 Target: x86_64-unknown-freebsd10.0 Thread model: posix > c++ test.cxx test.cxx:11:2: warning: "Doesn't have nullptr" [-W#warnings] #warning "Doesn't have nullptr" ^ 1 warning generated. > -- // Black Lion AKA Lev Serebryakov From dim at FreeBSD.org Fri Feb 7 15:47:51 2014 From: dim at FreeBSD.org (Dimitry Andric) Date: Fri, 7 Feb 2014 16:47:31 +0100 Subject: Why clang bundled with 10-RELEASE doesn't define __has_feature(cxx_nullptr)? In-Reply-To: <717736061.20140207174957@serebryakov.spb.ru> References: <717736061.20140207174957@serebryakov.spb.ru> Message-ID: <4D300C6C-4684-4C42-B059-4CC40258358F@FreeBSD.org> On 07 Feb 2014, at 14:49, Lev Serebryakov wrote: > Hello, freebsd-stable. > > Why __has_feature(cxx_nulptr) doesn't work on system compiler? nullptr > itself works! ... >> c++ test.cxx > test.cxx:11:2: warning: "Doesn't have nullptr" [-W#warnings] > #warning "Doesn't have nullptr" Hi Lev, You need to specify at least -std=c++0x for nullptr to be 'officially' available. -Dimitry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 203 bytes Desc: Message signed with OpenPGP using GPGMail URL: From lev at FreeBSD.org Fri Feb 7 16:34:26 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Fri, 7 Feb 2014 20:34:17 +0400 Subject: Is it possible to get make variable without recursive expansion? Message-ID: <12310461351.20140207203417@serebryakov.spb.ru> Hello, Freebsd-hackers. Suppose, I have in my Makefile such construction: A=xxx B=yyy ANB=${A} and ${B} Is it possible to get value of ANB without recursive expansion, i.e. string "${A} and ${B} and" in this case? I know, that all expansions are done at last moment (If I don't use ":=" operator), but is it possible to block second- and more-level expansion? I need to output "ANB=${ANB}" construction to external file without further expansion... -- // Black Lion AKA Lev Serebryakov From bright at mu.org Fri Feb 7 18:25:27 2014 From: bright at mu.org (Alfred Perlstein) Date: Fri, 07 Feb 2014 10:25:30 -0800 Subject: Is it possible to get make variable without recursive expansion? In-Reply-To: <12310461351.20140207203417@serebryakov.spb.ru> References: <12310461351.20140207203417@serebryakov.spb.ru> Message-ID: <52F5251A.8020401@mu.org> On 2/7/14 8:34 AM, Lev Serebryakov wrote: > Hello, Freebsd-hackers. > > > Suppose, I have in my Makefile such construction: > > A=xxx > B=yyy > ANB=${A} and ${B} > > Is it possible to get value of ANB without recursive expansion, i.e. string > "${A} and ${B} and" in this case? I know, that all expansions are done at > last moment (If I don't use ":=" operator), but is it possible to block > second- and more-level expansion? > > I need to output "ANB=${ANB}" construction to external file without further > expansion... > a: echo "AMB=\$${AMB}" ? -- Alfred Perlstein From lev at FreeBSD.org Fri Feb 7 18:52:14 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Fri, 7 Feb 2014 22:52:05 +0400 Subject: Is it possible to get make variable without recursive expansion? In-Reply-To: <52F5251A.8020401@mu.org> References: <12310461351.20140207203417@serebryakov.spb.ru> <52F5251A.8020401@mu.org> Message-ID: <201495029.20140207225205@serebryakov.spb.ru> Hello, Alfred. You wrote 7 ??????? 2014 ?., 22:25:30: AP> On 2/7/14 8:34 AM, Lev Serebryakov wrote: >> Hello, Freebsd-hackers. >> >> >> Suppose, I have in my Makefile such construction: >> >> A=xxx >> B=yyy >> ANB=${A} and ${B} >> >> Is it possible to get value of ANB without recursive expansion, i.e. string >> "${A} and ${B} and" in this case? I know, that all expansions are done at >> last moment (If I don't use ":=" operator), but is it possible to block >> second- and more-level expansion? >> >> I need to output "ANB=${ANB}" construction to external file without further >> expansion... >> AP> a: AP> echo "AMB=\$${AMB}" It will echo "AMB=${AMB}" as-is, as far as I understand, it is simple. And I need to get "ANB=${A} and ${B}" output. -- // Black Lion AKA Lev Serebryakov From bright at mu.org Fri Feb 7 19:07:47 2014 From: bright at mu.org (Alfred Perlstein) Date: Fri, 07 Feb 2014 11:07:50 -0800 Subject: Is it possible to get make variable without recursive expansion? In-Reply-To: <201495029.20140207225205@serebryakov.spb.ru> References: <12310461351.20140207203417@serebryakov.spb.ru> <52F5251A.8020401@mu.org> <201495029.20140207225205@serebryakov.spb.ru> Message-ID: <52F52F06.8080605@mu.org> On 2/7/14 10:52 AM, Lev Serebryakov wrote: > Hello, Alfred. > You wrote 7 ??????? 2014 ?., 22:25:30: > > AP> On 2/7/14 8:34 AM, Lev Serebryakov wrote: >>> Hello, Freebsd-hackers. >>> >>> >>> Suppose, I have in my Makefile such construction: >>> >>> A=xxx >>> B=yyy >>> ANB=${A} and ${B} >>> >>> Is it possible to get value of ANB without recursive expansion, i.e. string >>> "${A} and ${B} and" in this case? I know, that all expansions are done at >>> last moment (If I don't use ":=" operator), but is it possible to block >>> second- and more-level expansion? >>> >>> I need to output "ANB=${ANB}" construction to external file without further >>> expansion... >>> > AP> a: > AP> echo "AMB=\$${AMB}" > It will echo "AMB=${AMB}" as-is, as far as I understand, it is simple. And I need to get > "ANB=${A} and ${B}" output. > > > I do not think it's possible. You're basically asking "is there a way to ask Make for the text on the lhs of the variable assignment WITHOUT expansion? Your only option would be to "cheat" and not use the expanded form I think: A=aaa B=bbb AMB=\$${A} and \$${B} a: echo "AMB=${AMB}" Results in: .(19:07:30)(alfred at freefall.freebsd.org) ~ % make echo "AMB=\${A} and \${B}" AMB=${A} and ${B} .(19:07:30)(alfred at freefall.freebsd.org) ~ % There is no way to query make(1) (that I can see) for the exact value at the time of assignment and before expansion. -- Alfred Perlstein From lev at FreeBSD.org Fri Feb 7 19:38:53 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Fri, 7 Feb 2014 23:38:44 +0400 Subject: Is it possible to get make variable without recursive expansion? In-Reply-To: <52F52F06.8080605@mu.org> References: <12310461351.20140207203417@serebryakov.spb.ru> <52F5251A.8020401@mu.org> <201495029.20140207225205@serebryakov.spb.ru> <52F52F06.8080605@mu.org> Message-ID: <933525355.20140207233844@serebryakov.spb.ru> Hello, Alfred. You wrote 7 ??????? 2014 ?., 23:07:50: AP> Your only option would be to "cheat" and not use the expanded form I think: AP> A=aaa AP> B=bbb AP> AMB=\$${A} and \$${B} AP> a: AP> echo "AMB=${AMB}" AP> Results in: AP> .(19:07:30)(alfred at freefall.freebsd.org) AP> ~ % make AP> echo "AMB=\${A} and \${B}" AP> AMB=${A} and ${B} AP> .(19:07:30)(alfred at freefall.freebsd.org) AP> ~ % AP> There is no way to query make(1) (that I can see) for the exact value at AP> the time of assignment and before expansion. It is pity, because I need to make it with MAKE_ENV & CONFIGURE_MAKE variables in out port system :( -- // Black Lion AKA Lev Serebryakov From hrs at FreeBSD.org Fri Feb 7 19:46:56 2014 From: hrs at FreeBSD.org (Hiroki Sato) Date: Sat, 08 Feb 2014 04:45:59 +0900 (JST) Subject: Is it possible to get make variable without recursive expansion? In-Reply-To: <12310461351.20140207203417@serebryakov.spb.ru> References: <12310461351.20140207203417@serebryakov.spb.ru> Message-ID: <20140208.044559.1454108709341728013.hrs@allbsd.org> Lev Serebryakov wrote in <12310461351.20140207203417 at serebryakov.spb.ru>: le> Hello, Freebsd-hackers. le> le> le> Suppose, I have in my Makefile such construction: le> le> A=xxx le> B=yyy le> ANB=${A} and ${B} le> le> Is it possible to get value of ANB without recursive expansion, i.e. string le> "${A} and ${B} and" in this case? I know, that all expansions are done at le> last moment (If I don't use ":=" operator), but is it possible to block le> second- and more-level expansion? le> le> I need to output "ANB=${ANB}" construction to external file without further le> expansion... This may be what you want but works only with bmake: ----Makefile---- A=xxx B=yyy ANB=${A} and ${B} all: @echo ANB=$$(make -VANB) @echo ANB=$$(make .MAKE.EXPAND_VARIABLES=0 -VANB) ----Makefile---- -- Hiroki -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: not available URL: From dieterbsd at gmail.com Fri Feb 7 20:19:26 2014 From: dieterbsd at gmail.com (Dieter BSD) Date: Fri, 7 Feb 2014 12:19:23 -0800 Subject: opteron a1100 arm Message-ID: STefan writes, "parity" == "Gleichstand" ... Ya, ya, Ich verstehen. The parity-verses-ECC stuff was an (apparently failed) attempt at humor, hence the ":-)". Mark asks, > Is it even available to purchase? "AMD this quarter will ship the software development kits (CPUs, mainboards, etc.) to select software developers." "AMD Opteron A1100 "Seattle" system-on-chips will be commercially available in the second half of 2014." "The extremely popular GNU C compiler and the critical GNU C Library have already been ported to the 64-bit ARM architecture." The kit of course comes with penguinix. The bad news is that it has only 8 PCIe lanes. Are expansion cards considered uncool these days, or what? How are you supposed to add all the essential stuff they leave off the mainboard? http://www.xbitlabs.com/news/cpu/display/20140131110927_AMD_to_Ship_ARM_Based_Opteron_PCs_to_Software_Makers_This_Quarter.html (I'm still seeking info on Berlin. Hopefully the boards will have more than 1 or 2 expansion slots.) From bright at mu.org Fri Feb 7 20:44:41 2014 From: bright at mu.org (Alfred Perlstein) Date: Fri, 07 Feb 2014 12:44:37 -0800 Subject: Is it possible to get make variable without recursive expansion? In-Reply-To: <933525355.20140207233844@serebryakov.spb.ru> References: <12310461351.20140207203417@serebryakov.spb.ru> <52F5251A.8020401@mu.org> <201495029.20140207225205@serebryakov.spb.ru> <52F52F06.8080605@mu.org> <933525355.20140207233844@serebryakov.spb.ru> Message-ID: <52F545B5.7040407@mu.org> On 2/7/14, 11:38 AM, Lev Serebryakov wrote: > Hello, Alfred. > You wrote 7 ??????? 2014 ?., 23:07:50: > > AP> Your only option would be to "cheat" and not use the expanded form I think: > > AP> A=aaa > AP> B=bbb > AP> AMB=\$${A} and \$${B} > > AP> a: > AP> echo "AMB=${AMB}" > > AP> Results in: > AP> .(19:07:30)(alfred at freefall.freebsd.org) > AP> ~ % make > AP> echo "AMB=\${A} and \${B}" > AP> AMB=${A} and ${B} > AP> .(19:07:30)(alfred at freefall.freebsd.org) > AP> ~ % > > AP> There is no way to query make(1) (that I can see) for the exact value at > AP> the time of assignment and before expansion. > It is pity, because I need to make it with MAKE_ENV & CONFIGURE_MAKE > variables in out port system :( > > > ~ % make -f Makefile -XV AMB ? From lev at FreeBSD.org Fri Feb 7 21:12:33 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Sat, 8 Feb 2014 01:12:25 +0400 Subject: Is it possible to get make variable without recursive expansion? In-Reply-To: <52F545B5.7040407@mu.org> References: <12310461351.20140207203417@serebryakov.spb.ru> <52F5251A.8020401@mu.org> <201495029.20140207225205@serebryakov.spb.ru> <52F52F06.8080605@mu.org> <933525355.20140207233844@serebryakov.spb.ru> <52F545B5.7040407@mu.org> Message-ID: <173605890.20140208011225@serebryakov.spb.ru> Hello, Alfred. You wrote 8 ??????? 2014 ?., 0:44:37: AP> ~ % make -f Makefile -XV AMB Yoohoo! It works! -- // Black Lion AKA Lev Serebryakov From cary at sdf.org Sat Feb 8 01:21:49 2014 From: cary at sdf.org (Cary) Date: Fri, 07 Feb 2014 17:21:39 -0800 Subject: remove shutdown messages in console In-Reply-To: <1D565840-1ADA-4567-85B0-FB4AE771B823@bsdimp.com> References: <52F05DFC.7070007@freebsd.org> <1D565840-1ADA-4567-85B0-FB4AE771B823@bsdimp.com> Message-ID: <52F586A3.6080404@sdf.org> Warner Losh wrote: > > On Feb 3, 2014, at 8:26 PM, Julian Elischer wrote: > >> On 2/2/14, 6:07 PM, Reza Mirzazadeh wrote: >>> hi . how can i remove shutdown messages in console ? or even print my >>> arbitrary messages in console ? >>> >>> >>> *** FINAL System shutdown message from root at geekiam *** >>> System going down IMMEDIATELY >>> >>> some messages like these ! i don't want it print theses messages >>> -- >>> _______________________________________________ >>> freebsd-hackers at freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" >>> >> we implimented the consmute sysctl option to "mute" an otherwise configured console for exactly this purpose. >> (and boot messages). > > Except these are different beast. They aren't kernel messages, but generated by userland... > > to get rid of these, hack /etc/syslog.conf to get rid of the *.emerg * line. > Or hack shutdown to stop sending them.... > Or use 'halt' and 'reboot' only > > Warner > > > It was easy to substitute an arbitrary message. I used the following command: # sed 's/System going down IMMEDIATELY/Farewell, Doctor F, until our paths meet yet once more/' /usr/src/sbin/shutdown/shutdown.c >new.shutdown.c then ran "clang -o shutdown new.shutdown.c" . If the file "shutdown" were copied to /sbin would that work as expected? Cary -- cary at sdf.org SDF Public Access UNIX System - http://sdf.org ------------------------------ From rmacklem at uoguelph.ca Mon Feb 10 06:01:11 2014 From: rmacklem at uoguelph.ca (Rick Macklem) Date: Mon, 10 Feb 2014 01:01:10 -0500 (EST) Subject: threads getting stuck in vmem_bt_alloc() at "btalloc"? Message-ID: <1714012824.3160193.1392012070166.JavaMail.root@uoguelph.ca> Hi, I've been doing some testing using pagesize clusters (4K) for NFS instead of mclbytes (2K) on a single core i386. Sometimes I get threads stuck sleeping on "btalloc", which seems to happen in vmem_bt_alloc(). The comment in vmem_bt_alloc() basically says: out of address space or lost a fill race Since this is persistent, I suspect it is the first case? So, does anyone know what is going on here or what I should look at to try and resolve this? Btw, when I am testing, I don't see the pagesize cluster allocation exceed 400, so it doesn't seem to be a leak or excessive allocation. Thanks in advance for any help, rick ps: Garrett, I cc'd you in part because I remember you mentioning something about mclbyte clusters being pre-allocated vs ?? Maybe, you could explain what you were talking about, since I didn't follow it the last time. From wojtek at wojtek.tensor.gdynia.pl Mon Feb 10 10:52:26 2014 From: wojtek at wojtek.tensor.gdynia.pl (Wojciech Puchar) Date: Mon, 10 Feb 2014 11:42:36 +0100 (CET) Subject: opteron a1100 arm In-Reply-To: References: Message-ID: > > The bad news is that it has only 8 PCIe lanes. Are expansion cards > considered uncool these days, or what? How are you supposed to add all > the essential stuff they leave off the mainboard? anyway - it already have 10Gb/s ethernet, USB and 8 SATA ports. no need for high speed I/O expansions. and 8 single lane PCIe ports seems enough for me. From egoitz at ramattack.net Tue Feb 11 18:13:10 2014 From: egoitz at ramattack.net (Egoitz Aurrekoetxea) Date: Tue, 11 Feb 2014 19:05:26 +0100 Subject: mbuf_tag memory freeing issues with LRO enabled on the XENHVM driver Message-ID: <1AECA876-08BC-48F8-B356-42CE35100805@ramattack.net> Good afternoon, It seems the LRO code inside the FreeBSD's kernel is causing the first mbuf chain?s anymbuf->m_hdr->mh_flags due to some failure not to contain the M_PKTHDR flag. Have you ever seen this bad behavior with any of these drivers having LRO enabled : cxgb cxgbe e1000 ixgbe ixgbe mxge oce qlxgb qlxge virtio vxge The no presence of this flag, seems to be causing the kernel not to be able to enter : static void mb_dtor_pack(void *mem, int size, void *arg) { struct mbuf *m; m = (struct mbuf *)mem; if ((m->m_flags & M_PKTHDR) != 0) m_tag_delete_chain(m, NULL); <============ to this function. causing the wired memory to increase a lot due mbuf_tags usage memory to be pretty high. I have noticed about this issue using the XENHVM network driver (xn), but taking a look at some other drivers using the same code as this one? have found the commented before ones?. Has anyone too suffered the issue?. Best regards, From wojtek at wojtek.tensor.gdynia.pl Tue Feb 11 18:36:23 2014 From: wojtek at wojtek.tensor.gdynia.pl (Wojciech Puchar) Date: Tue, 11 Feb 2014 19:36:04 +0100 (CET) Subject: FreeBSD UDF support Message-ID: what is the status? i see mount_udf and UDF filesystem in kernel options. man mount_udf doesn't state it's read only - so i assume RW support is there. but no newfs_udf tried to compile udfclient from ports - doesn't compile, both clang and gcc FreeBSD 10 about month old from cvs. any ideas? From jhb at freebsd.org Tue Feb 11 19:49:42 2014 From: jhb at freebsd.org (John Baldwin) Date: Tue, 11 Feb 2014 13:19:14 -0500 Subject: Atom Board ACPI API MOPNV10J failing since 9.1 In-Reply-To: <52DED060.4070800@yahoo.com> References: <52CF850A.9060906@erdgeist.org> <201401211215.22021.jhb@freebsd.org> <52DED060.4070800@yahoo.com> Message-ID: <201402111319.14546.jhb@freebsd.org> On Tuesday, January 21, 2014 2:54:08 pm David Shane Holden wrote: > On 01/21/14 12:15, John Baldwin wrote: > > > > Hmm, I think I see the issue, and I might have a fix for it in the > > works already. The problem is that we haven't reserved pcib1's > > windows when agp probes, so agp0 steals a resource that is already > > in use. The change I have that might fix this isn't trivial though, > > so I don't have a patch I can just give you to test right now. :( > > Also, this isn't a BIOS issue per se. > > > > I came to the same conclusion a few days ago when I started digging into > it more. The agp driver requests some memory and the resource manager > gives it a chunk which should be reserved for the bridge. I couldn't > find an elegant solution; the best I could come up with was to add the > bridge devices first in pci_add_children() to ensure they got attached > before anything else. Which worked, but seems like a hack. Feel free > to ping me if you have something that needs testing. The patch I have in mind is a more general version of this that would have the same effect. I haven't tested it in quite a while, but if you are brave you can test it. http://people.freebsd.org/~jhb/patches/multipass.patch -- John Baldwin From jhb at freebsd.org Tue Feb 11 19:49:48 2014 From: jhb at freebsd.org (John Baldwin) Date: Tue, 11 Feb 2014 13:30:54 -0500 Subject: The sonewconn listen queue overflow issue In-Reply-To: <52EFEEF7.5010704@xs4all.nl> References: <52EFEEF7.5010704@xs4all.nl> Message-ID: <201402111330.54758.jhb@freebsd.org> On Monday, February 03, 2014 2:33:11 pm Photo stuff wrote: > L.S. > > I came across a lot of messages in the log since, I think, about 9.1, > that sometimes fill it up so much that all else is made invisible, of > the type: > > sonewconn: pcb 0xyyyyyyyyyyyyyyyy: Listen queue overflow: 8 already in > queue awaiting acceptance > > I searched a bit on the web and came across recommendations to try > netstat -nAa to find out which program this came from. > > Well, running netstat -nAa |grep pcb 0xyyyyyyyyyyyyyyyy in a loop didn't > work, it didn't give any output even though the messages kept coming in > the log during that time. > > I forgot about it then recently searched a bit more, and I found this page: > > http://lawrencechen.net/2014/sonewconn-pcb-0xfffffe006acd9310-listen-queue > > So a listen queue overflow of 8 is caused by a listen value of 5 (QLEN > > 3 * (QLIM / 2)) > > Doing: > netstat -LaAn | grep '/5 ' > > resulted in > ffffzzzzzzzzzzzz tcp4 0/0/5 127.0.0.1:8000 > > The ffffzzz value was not the value in the message log, > but the connection means in my case it had to be the junkbuster proxy > (which I still use, still works well :) ), as I didn't run anything else > locally. > > So I looked into junkbuster's cource and in bind.c I changed 2 instances of: > > while (listen(fd, 5) == -1) { > > to > > while (listen(fd, 10) == -1) { /* 10 instead of 5, this fixes dmesg > spamming of the type 'sonewconn: pcb 0xyyyyyyyyyyyyyyyy: Listen queue > overflow: 8 already in queue awaiting acceptance'? */ > > This improved the situation, but still gave the issue of the log filling > up too much. So then I changed it to 20, which gave me silence :) > > Checking the latest log I couldn't find anything in the last 10 days. > > So for people who have this issue I recommend calculating which value > the listen-value that the overflow-value corresponds to, then checking > as above and then it should be possible to find the daemon causing the > issue. And modify that program... > > But while this removes the errors, what do these messages really > signify? I mean which didn't this happen before in earlier versions of > FreeBSD? Maybe earlier versions just dropped the connections without logging a message? The message means that connections are arriving faster than the userland program can accept them. There is a stat for that in 'netstat -s -p tcp' and you can see if it was increasing on older versions of FreeBSD (if you still have a machine with that around) to make sure the only change is the additional log message. > Btw., at the moment (running 10.0 RC2) my message log now gets filled up > with something else :) > > hwpstate0: set freq failed, err 6 > > This happened long ago already in 9.1 with my AMD3500+ and still now on > my AMD FX6100... I am not familiar with how hwpstate works, sorry. -- John Baldwin From jhb at freebsd.org Tue Feb 11 19:49:51 2014 From: jhb at freebsd.org (John Baldwin) Date: Tue, 11 Feb 2014 13:38:41 -0500 Subject: FreeBSD UDF support In-Reply-To: References: Message-ID: <201402111338.41451.jhb@freebsd.org> On Tuesday, February 11, 2014 1:36:04 pm Wojciech Puchar wrote: > what is the status? > > i see mount_udf and UDF filesystem in kernel options. man mount_udf > doesn't state it's read only - so i assume RW support is there. No, it is read-only. -- John Baldwin From jhb at freebsd.org Tue Feb 11 19:49:44 2014 From: jhb at freebsd.org (John Baldwin) Date: Tue, 11 Feb 2014 13:21:02 -0500 Subject: core dump vs kern.ipc.shm_use_phys In-Reply-To: <52DFAC31.6030905@FreeBSD.org> References: <52DFAC31.6030905@FreeBSD.org> Message-ID: <201402111321.02294.jhb@freebsd.org> On Wednesday, January 22, 2014 6:32:01 am Andriy Gapon wrote: > > I seems that if kern.ipc.shm_use_phys is enabled then shared memory regions are > not included into a coredump. Seems that each_writable_segment() in > sys/kern/imgact_elf.c skips OBJT_PHYS objects. Hmm, that may be a feature. I often map large shared memory segments with MAP_NOCORE on purpose. Is it to tell if a given OBJT_PHYS object is a SYSV SHM object? (I assume it isn't.) -- John Baldwin From jhb at freebsd.org Tue Feb 11 19:49:47 2014 From: jhb at freebsd.org (John Baldwin) Date: Tue, 11 Feb 2014 13:25:24 -0500 Subject: ULE locking mechanism In-Reply-To: References: Message-ID: <201402111325.24523.jhb@freebsd.org> On Tuesday, January 28, 2014 8:07:08 am Jens Krieg wrote: > Hello, > > we are currently working on project for our university. Our goal is to implement a simple round robin scheduler for FreeBSD 9.2 on a single core machine. > So far we removed most of the functionality of the ULE scheduler except the functions that are called from outside. The system successfully boots to user land with our RR scheduler managing thread in a list based run queue. Further, it is possible to interact with the system using the shell. > > The next step is to replace the locking mechanism of the ULE scheduler. Therefore, we replaced the scheduling dependent thread_lock/thread_unlock functions by simply disabling/enabling the interrupts. With this modification the kernel works fine until we hit the user land then the system crashes. > The error occurs in the init user process (init_main.c:start_init:685). We found out that the page fault is triggered while executing the subyte function for the first time. See the error description below (unfortunately not shown in backtrace). > We compared the ULE scheduler with our RR implementation and it appears, that the parameters passed to subyte as well as the register values are identical. We assume, that whatever caused the error is related to the thread locking replacement. > > Every time the kernel want to modify thread data the corresponding thread is locked to prevent any interference by other threads. Since we are using a single core machine why isn?t it sufficient to simply disable interrupt while modifying thread data. Could you provide us with detailed information about the locking mechanism in FreeBSD and also answer the following questions, please. > > What is the purpose of thread_lock/thread_unlock besides protecting thread data? > How does the TDQ LOCK works and how is it related to a thread LOCK? > - all thread LOCKs of the thread located in the run queue pointing to the TDQ LOCK, and > - the TDQ LOCK points to the currently running thread > - on context switching the current thread passes the TDQ LOCK to the new chosen thread > - Could you explain the idea behind that locking concept, please? > Any suggestions we shall care about in our own lock implementation? thread_lock is quite intertwined with other locks. E.g. when a thread is blocked on a turnstile, thread_lock() for that thread locks the 'ts_lock' spin mutex for that turnstile. If you want to replace thread lock, you need to change all the locks that td_lock can be to use your new primitive. You'd probably have an easier time just changing how mtx_lock_spin() works. (In fact, if you just disable 'options SMP', the stock kernel turns mtx_lock_spin() into a function that just disables interrupts.) For your core dump, the first step would be to use gdb to map that address to a file line. For example, you can just do 'l *fork_exit+0x9d', or you can do 'l *' where you use the value from the trap message. Looking at that can probably tell you why you panic'd. -- John Baldwin From rhurlin at gwdg.de Tue Feb 11 21:23:15 2014 From: rhurlin at gwdg.de (Rainer Hurling) Date: Tue, 11 Feb 2014 22:23:11 +0100 Subject: FreeBSD UDF support In-Reply-To: References: Message-ID: <52FA94BF.80304@gwdg.de> Am 11.02.2014 19:36, schrieb Wojciech Puchar: > what is the status? > > i see mount_udf and UDF filesystem in kernel options. man mount_udf > doesn't state it's read only - so i assume RW support is there. > > but no newfs_udf > > tried to compile udfclient from ports - doesn't compile, both clang and gcc > > FreeBSD 10 about month old from cvs. > > any ideas? Not a direct answer on your question with base mount_udf. Do you know https://github.com/williamdevries/UDF/ from William DeVries? His project comes from NetBSD and modernises the UDF standard towards 2.5x. With this driver, you would be able to read modern not encrypted Blu-rays and DVDs. The driver works in principle, some minor issues have to be solved. Unfortunately there is only very little interest to integrate this driver into base. Regards, Rainer From dpejesh at yahoo.com Tue Feb 11 22:12:25 2014 From: dpejesh at yahoo.com (David Shane Holden) Date: Tue, 11 Feb 2014 17:12:11 -0500 Subject: Atom Board ACPI API MOPNV10J failing since 9.1 In-Reply-To: <201402111319.14546.jhb@freebsd.org> References: <52CF850A.9060906@erdgeist.org> <201401211215.22021.jhb@freebsd.org> <52DED060.4070800@yahoo.com> <201402111319.14546.jhb@freebsd.org> Message-ID: On 02/11/14 13:19, John Baldwin wrote: > > The patch I have in mind is a more general version of this that > would have the same effect. > > I haven't tested it in quite a while, but if you are brave you can > test it. > > http://people.freebsd.org/~jhb/patches/multipass.patch > When I tested your pci_bus_rman3.patch the other day I ran it on this Atom board. To my surprise the problem seemed to have been fixed somewhere along the line. I'm not sure if it was your patch, r261243, some other commit, or me just not having it configured properly to trigger the problem again. I'll try to spend some time tonight on it. From adrian at freebsd.org Tue Feb 11 23:35:16 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Tue, 11 Feb 2014 15:35:15 -0800 Subject: mbuf_tag memory freeing issues with LRO enabled on the XENHVM driver In-Reply-To: <1AECA876-08BC-48F8-B356-42CE35100805@ramattack.net> References: <1AECA876-08BC-48F8-B356-42CE35100805@ramattack.net> Message-ID: +net .. any ideas? -a On 11 February 2014 10:05, Egoitz Aurrekoetxea wrote: > Good afternoon, > > It seems the LRO code inside the FreeBSD's kernel is causing the first mbuf chain's anymbuf->m_hdr->mh_flags due to some failure not to contain the M_PKTHDR flag. > Have you ever seen this bad behavior with any of these drivers having LRO enabled : > > cxgb > cxgbe > e1000 > ixgbe > ixgbe > mxge > oce > qlxgb > qlxge > virtio > vxge > > The no presence of this flag, seems to be causing the kernel not to be able to enter : > > static void > mb_dtor_pack(void *mem, int size, void *arg) > { > struct mbuf *m; > > m = (struct mbuf *)mem; > if ((m->m_flags & M_PKTHDR) != 0) > m_tag_delete_chain(m, NULL); <============ to this function. > > causing the wired memory to increase a lot due mbuf_tags usage memory to be pretty high. > > I have noticed about this issue using the XENHVM network driver (xn), but taking a look at some other drivers using the same code as this one... have found the commented before ones.... > > Has anyone too suffered the issue?. > > Best regards, > > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" From Suresh.Gumpula at netapp.com Wed Feb 12 02:02:08 2014 From: Suresh.Gumpula at netapp.com (Gumpula, Suresh) Date: Wed, 12 Feb 2014 02:02:01 +0000 Subject: malloc(9) and its alignment Message-ID: Hi, It appears the malloc(9) returns 8 byte aligned ( UMA_ALIGN_PTR) pointers, but in bus_dmamem_alloc we might end up checking for greater alignment if we take malloc(9) path instead contig_malloc. Can someone please confirm if malloc(9) returns different alignment pointers ? bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags, bus_dmamap_t *mapp) { /* * XXX: * (dmat->alignment < dmat->maxsize) is just a quick hack; the exact * alignment guarantees of malloc need to be nailed down, and the * code below should be rewritten to take that into account. * * In the meantime, we'll warn the user if malloc gets it wrong. */ if ((dmat->maxsize <= PAGE_SIZE) && (dmat->alignment < dmat->maxsize) && dmat->lowaddr >= ptoa((vm_paddr_t)Maxmem)) { *vaddr = malloc(dmat->maxsize, M_DEVBUF, mflags); } else { *vaddr = contigmalloc(dmat->maxsize, M_DEVBUF, mflags, 0ul, dmat->lowaddr, dmat->alignment? dmat->alignment : 1ul, dmat->boundary); } if (vtophys(*vaddr) & (dmat->alignment - 1)) { NETAPP_MUTED_PRINTF("bus_dmamem_alloc failed to align memory properly.\n"); Regards, Suresh From jhellenthal at dataix.net Wed Feb 12 05:09:35 2014 From: jhellenthal at dataix.net (Jason Hellenthal) Date: Wed, 12 Feb 2014 00:09:34 -0500 Subject: Thoughts on Multi-Symlink Concept Message-ID: Hi All, Now I already know the thoughts on symlink hell within filesystems all to well considering most Linux flavoring's. But I am curious as to what all your opinions would be to add symlink support to multiple target files much like what you could do with cat(1) or portalfs to include a bunch of files in one instance but similar to the following examples in place of such. Instead of: cat /path/to/files* ln -sm /path/to/files* ./my_concat_list cat ./my_concat_list Or ln -sm /path/to/file1 /path/to/file2 ./my_concat_filters pfctl -v -f ./my_concat_filters Personally while I know it's a hack, but I feel it would bring some glue to programs and other such situations that do not have file include support and add support per-say way to create a repeatable playlist to shorten user operations at any given time. Obviously this isn't anywhere else implemented and would need to be a BSD extension of ln(1) but I find that it could be a beneficial feature for those that could use it to its full potential. I've thought about the same instance also being done with hardlinks but I keep coming across the idea that there are too many race conditions that would be found with that. Anyway . . . opinions, thoughts, ideas, criticism . . . welcome. Thanks for your time. From george+freebsd at m5p.com Wed Feb 12 11:08:34 2014 From: george+freebsd at m5p.com (George Mitchell) Date: Wed, 12 Feb 2014 06:08:26 -0500 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: Message-ID: <52FB562A.20509@m5p.com> On 02/12/14 00:09, Jason Hellenthal wrote: > Hi All, > > Now I already know the thoughts on symlink hell within filesystems all to > well considering most Linux flavoring's. But I am curious as to what all > your opinions would be to add symlink support to multiple target files much > like what you could do with cat(1) or portalfs to include a bunch of files > in one instance but similar to the following examples in place of such. > > Instead of: cat /path/to/files* > ln -sm /path/to/files* ./my_concat_list > cat ./my_concat_list > > Or > > ln -sm /path/to/file1 /path/to/file2 ./my_concat_filters > pfctl -v -f ./my_concat_filters > > Personally while I know it's a hack, but I feel it would bring some glue to > programs and other such situations that do not have file include support > and add support per-say way to create a repeatable playlist to shorten user > operations at any given time. > > Obviously this isn't anywhere else implemented and would need to be a BSD > extension of ln(1) but I find that it could be a beneficial feature for > those that could use it to its full potential. > > I've thought about the same instance also being done with hardlinks but I > keep coming across the idea that there are too many race conditions that > would be found with that. > > Anyway . . . opinions, thoughts, ideas, criticism . . . welcome. > > > Thanks for your time. [...] My gut feeling is that it pushes functionality that belongs in userland down into the kernel, plus probably introducing new security loopholes. -- George From ian at FreeBSD.org Wed Feb 12 14:19:54 2014 From: ian at FreeBSD.org (Ian Lepore) Date: Wed, 12 Feb 2014 07:19:48 -0700 Subject: malloc(9) and its alignment In-Reply-To: References: Message-ID: <1392214788.1145.52.camel@revolution.hippie.lan> On Wed, 2014-02-12 at 02:02 +0000, Gumpula, Suresh wrote: > Hi, > It appears the malloc(9) returns 8 byte aligned ( UMA_ALIGN_PTR) pointers, but in bus_dmamem_alloc we might end up checking for greater alignment > if we take malloc(9) path instead contig_malloc. > Can someone please confirm if malloc(9) returns different alignment pointers ? > > bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags, > bus_dmamap_t *mapp) > { > /* > * XXX: > * (dmat->alignment < dmat->maxsize) is just a quick hack; the exact > * alignment guarantees of malloc need to be nailed down, and the > * code below should be rewritten to take that into account. > * > * In the meantime, we'll warn the user if malloc gets it wrong. > */ > if ((dmat->maxsize <= PAGE_SIZE) && > (dmat->alignment < dmat->maxsize) && > dmat->lowaddr >= ptoa((vm_paddr_t)Maxmem)) { > *vaddr = malloc(dmat->maxsize, M_DEVBUF, mflags); > } else { > > *vaddr = contigmalloc(dmat->maxsize, M_DEVBUF, mflags, > 0ul, dmat->lowaddr, dmat->alignment? dmat->alignment : 1ul, > dmat->boundary); > } > if (vtophys(*vaddr) & (dmat->alignment - 1)) { > NETAPP_MUTED_PRINTF("bus_dmamem_alloc failed to align memory properly.\n"); > > Regards, > Suresh In my experience, the effective malloc(9) alignment ends up being the same as MINALLOCSIZE, which is UMA_SMALLEST_UNIT, which is 16 bytes on a system with 4K pages. At $work we overrode MINALLOCSIZE to 32 to work around cache line alignment problems in busdma for ARM systems a few years ago. There is a newer set of busdma allocator routines available in kern/subr_busdma_bufalloc.c which are designed to give a platform more control over alignment of busdma buffers smaller than a page. An example of using them can be found in arm/busdma_machdep[-v6].c. As far as I know, they're only being used on ARM platforms right now. -- Ian From Suresh.Gumpula at netapp.com Wed Feb 12 19:40:18 2014 From: Suresh.Gumpula at netapp.com (Gumpula, Suresh) Date: Wed, 12 Feb 2014 19:40:15 +0000 Subject: malloc(9) and its alignment In-Reply-To: <1392214788.1145.52.camel@revolution.hippie.lan> References: <1392214788.1145.52.camel@revolution.hippie.lan> Message-ID: Thanks Ian for the reply. I will look at the ARM code, but I was thinking why malloc(9) does not return bucket size aligned pointers. Regards, Suresh -----Original Message----- From: Ian Lepore [mailto:ian at FreeBSD.org] Sent: Wednesday, February 12, 2014 9:20 AM To: Gumpula, Suresh Cc: freebsd-hackers at freebsd.org Subject: Re: malloc(9) and its alignment On Wed, 2014-02-12 at 02:02 +0000, Gumpula, Suresh wrote: > Hi, > It appears the malloc(9) returns 8 byte aligned ( UMA_ALIGN_PTR) > pointers, but in bus_dmamem_alloc we might end up checking for greater alignment if we take malloc(9) path instead contig_malloc. > Can someone please confirm if malloc(9) returns different alignment pointers ? > > bus_dmamem_alloc(bus_dma_tag_t dmat, void** vaddr, int flags, > bus_dmamap_t *mapp) > { > /* > * XXX: > * (dmat->alignment < dmat->maxsize) is just a quick hack; the exact > * alignment guarantees of malloc need to be nailed down, and the > * code below should be rewritten to take that into account. > * > * In the meantime, we'll warn the user if malloc gets it wrong. > */ > if ((dmat->maxsize <= PAGE_SIZE) && > (dmat->alignment < dmat->maxsize) && > dmat->lowaddr >= ptoa((vm_paddr_t)Maxmem)) { > *vaddr = malloc(dmat->maxsize, M_DEVBUF, mflags); > } else { > > *vaddr = contigmalloc(dmat->maxsize, M_DEVBUF, mflags, > 0ul, dmat->lowaddr, dmat->alignment? dmat->alignment : 1ul, > dmat->boundary); > } > if (vtophys(*vaddr) & (dmat->alignment - 1)) { > NETAPP_MUTED_PRINTF("bus_dmamem_alloc failed to align > memory properly.\n"); > > Regards, > Suresh In my experience, the effective malloc(9) alignment ends up being the same as MINALLOCSIZE, which is UMA_SMALLEST_UNIT, which is 16 bytes on a system with 4K pages. At $work we overrode MINALLOCSIZE to 32 to work around cache line alignment problems in busdma for ARM systems a few years ago. There is a newer set of busdma allocator routines available in kern/subr_busdma_bufalloc.c which are designed to give a platform more control over alignment of busdma buffers smaller than a page. An example of using them can be found in arm/busdma_machdep[-v6].c. As far as I know, they're only being used on ARM platforms right now. -- Ian From wkoszek at freebsd.org Wed Feb 12 21:52:15 2014 From: wkoszek at freebsd.org (Wojciech A. Koszek) Date: Wed, 12 Feb 2014 21:51:23 +0000 Subject: FreeBSD GSOC 2014: call for ideas! Message-ID: <20140212215123.GB31185@FreeBSD.org> (cross-posted message; keep discussion on hackers@ only) Hello, We want to participate in GSOC 2014. We need more ideas for students, who can be new to FreeBSD and Open Source. Submit your ideas here: http://tinyurl.com/FreeBSD-GSOC2014 If you make them attractive and clearly explained, there should be a bigger chance of getting them picked by somebody. All submitted ideas upon review will be put here: https://wiki.freebsd.org/SummerOfCode2014 If you have a Wiki access, please just copy the skeleton from the top of this page and use it for your task. Thanks, -- Wojciech A. Koszek wkoszek at FreeBSD.org http://www.koszek.com/ From jmg at funkthat.com Wed Feb 12 22:07:13 2014 From: jmg at funkthat.com (John-Mark Gurney) Date: Wed, 12 Feb 2014 14:07:05 -0800 Subject: malloc(9) and its alignment In-Reply-To: References: <1392214788.1145.52.camel@revolution.hippie.lan> Message-ID: <20140212220705.GV34851@funkthat.com> Gumpula, Suresh wrote this message on Wed, Feb 12, 2014 at 19:40 +0000: > Thanks Ian for the reply. I will look at the ARM code, but I was thinking why malloc(9) does not return bucket size aligned pointers. Always returning bucket sizes aligned pointers may not be ideal for a cache.. say you have a buffer of 512 bytes, where often only the first 128 bytes are used (but all 512 bytes may be)... If you always align at 512, some cache lines will be more heavily used than others, reducing the effective size of the cache... This is only one reason not aligning to size may be better... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From rpokala at panasas.com Thu Feb 13 01:53:52 2014 From: rpokala at panasas.com (Pokala, Ravi) Date: Thu, 13 Feb 2014 01:51:48 +0000 Subject: AF-4Kn and DEV_BSIZE Message-ID: Hi hackers, At Panasas, we're working on supporting AF-4Kn drives (and also optimizing for AF-512e drives while we're at it) in our legacy codebase, which is based on: 7.2-RELEASE - a bunch of drivers we don't need - a bunch of utilities we don't use + a bunch of bug fixes merged from 7-STABLE + a bunch of drivers back-ported from {8,9,10}-STABLE + a bunch of internal changes Since 7-STABLE doesn't support AF-4Kn, I'm looking at 10-STABLE for inspiration. We're only interested in SATA, so this basically breaks down into: ata(4) / ad(4) [1] GEOM [2] General kernel stuff I have a pretty good handle on the drivers and GEOM, but I'd like to discuss the "General kernel stuff" bucket. Of particular concern is DEV_BSIZE - and it's relatives: DEV_BSHIFT, btodb() and dbtob() - which are used pretty much everywhere. A bunch of that is in places that don't come anywhere near the drives, so I'm not worried about them. But when I see those macros used in places that *do* come near the drives, I get nervous. For example: sys/kern/vfs_aio.c sys/kern/vfs_bio.c sys/kern/kern_physio.c sys/vm/swap_pager.c sys/vm/vnode_pager.c Swapping, paging, and block IO all come near the drives. On the one hand, the fact that we're doing things in terms of 512-byte blocks all those places suggests that lots of stuff has to change to work with 4KB drive blocks. On the other hand, when I compare the use of those macros in our code base against what's in 10-STABLE, they are substantively the same; since 10-STABLE supports AF-4Kn, that implies it's okay to use those macros those ways. But I'm fuzzy on *how*, if we're passing around 512-byte buffers, everything still works okay with AF-4Kn drives. Can someone shed some light? Thanks, Ravi [1] The elephant in the room was the move from the old ATA driver to ATACAM. Fortunately, the code in sys/cam/ata/ata_da.c::adaregister() that maps from the logical and physical sector sizes detected from the drive, to the more abstract 'struct disk' used by GEOM, is pretty easy to understand and shoehorn into the 7-ish drivers. Once that's done and the disk softc is carrying around the logical sector size, then a bunch of places get changed to use that value instead of DEV_BSIZE. There are a few other nits, but that's the bulk of the changes needed for ata(4) / ad(4). [2] geom_disk.c::g_disk_create() copies 'struct disk's 'd_sectorsize' and 'd_stripesize' into the provider's 'sectorsize' and 'stripesize'. The rest of the GEOM stack uses those correctly, so there aren't any real changes required in GEOM. Hurray! --rp From jordan.hubbard at gmail.com Thu Feb 13 05:08:34 2014 From: jordan.hubbard at gmail.com (Jordan Hubbard) Date: Wed, 12 Feb 2014 21:08:30 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: Message-ID: On Feb 11, 2014, at 9:09 PM, Jason Hellenthal wrote: > Instead of: cat /path/to/files* > ln -sm /path/to/files* ./my_concat_list > cat ./my_concat_list > > Or > > ln -sm /path/to/file1 /path/to/file2 ./my_concat_filters > pfctl -v -f ./my_concat_filters Globbing is done in user land (by the shell) - you wouldn?t want to push that down into the kernel, which is either what you?d have to do or you?d need a user land daemon which did round-trips with the kernel to do the translation, which would also need to make sure to get all of the process permission stuff right since the user id / gid / $CWD would all potentially affect the expansion of the ?symlink?. There are other problems. Since ?my_concat_list? now expands to multiple files, you have to pre-expand its contents before actually execing cat(1), and since filenames can have spaces and other magic characters (to the shell) in them, you would need to be able to deal with escaping all of that. Just like? the shell does! Even worse still, you?d be essentially creating a new file type with very non-file like semantics. Even variant symlinks (/bin -> /${ARCH}/bin), which can expand differently depending on the user context, have clearly understandable semantics - you know that the symlink is going to expand to exactly one file no matter what ARCH is set to. How, conversely, would you expect ?cat < my_concat_filters? or ?dd if=my_concat_filters of=?? to work? You?ve essentially created a construct that can cause truly weird behaviors unless you use it in exactly one way and one way only (as a file argument list), the user having to explicitly know the difference when they consume the symlink. That creates a whole raft of layering violations, and is very ?not Unix?, to be honest. If you want to get creative along these lines without breaking Unix, I think creating some new type of artificial directory that doesn?t actually physically contain files but can flexibly reference a collection of files based on some sort of query - that might be interesting. - Jordan From jordan.hubbard at gmail.com Thu Feb 13 05:17:23 2014 From: jordan.hubbard at gmail.com (Jordan Hubbard) Date: Wed, 12 Feb 2014 21:17:19 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: Message-ID: On Feb 12, 2014, at 9:08 PM, Jordan Hubbard wrote: > Globbing is done in user land (by the shell) - you wouldn?t want to push that down into the kernel, which is either what you?d have to do or you?d need a user land daemon which did round-trips with the kernel to do the translation, which would also need to make sure to get all of the process permission stuff right since the user id / gid / $CWD would all potentially affect the expansion of the ?symlink?. Actually, just to correct myself, there is a third way, which is that you could make the shell also do the expansion of the symlink (or interpose it into libc), but now you?d just be stacking one weird hack on top of another weird hack. It?s still not a good idea for all the reasons I mentioned, at least not as a ?symlink?. Maybe some new type of shell builtin, though I?m not sure how/where you?d use it. - Jordan From bakul at bitblocks.com Thu Feb 13 06:41:12 2014 From: bakul at bitblocks.com (Bakul Shah) Date: Wed, 12 Feb 2014 22:35:01 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: Message-ID: On Feb 12, 2014, at 9:08 PM, Jordan Hubbard wrote: > > On Feb 11, 2014, at 9:09 PM, Jason Hellenthal wrote: > >> Instead of: cat /path/to/files* >> ln -sm /path/to/files* ./my_concat_list >> cat ./my_concat_list >> >> Or >> >> ln -sm /path/to/file1 /path/to/file2 ./my_concat_filters >> pfctl -v -f ./my_concat_filters > > Globbing is done in user land (by the shell) - you wouldn?t want to push that down into the kernel, which is either what you?d have to do or you?d need a user land daemon which did round-trips with the kernel to do the translation, which would also need to make sure to get all of the process permission stuff right since the user id / gid / $CWD would all potentially affect the expansion of the ?symlink?. > > There are other problems. Since ?my_concat_list? now expands to multiple files, you have to pre-expand its contents before actually execing cat(1), and since filenames can have spaces and other magic characters (to the shell) in them, you would need to be able to deal with escaping all of that. Just like? the shell does! > > Even worse still, you?d be essentially creating a new file type with very non-file like semantics. Even variant symlinks (/bin -> /${ARCH}/bin), which can expand differently depending on the user context, have clearly understandable semantics - you know that the symlink is going to expand to exactly one file no matter what ARCH is set to. How, conversely, would you expect ?cat < my_concat_filters? or ?dd if=my_concat_filters of=?? to work? You?ve essentially created a construct that can cause truly weird behaviors unless you use it in exactly one way and one way only (as a file argument list), the user having to explicitly know the difference when they consume the symlink. That creates a whole raft of layering violations, and is very ?not Unix?, to be honest. May be a similar idea can be used to implement file striping & mirroring on top of existing filesystems -- modern distributed FS seem to reinvent a lot stuff. Instead one can just build a simple thin layer and use local and remote simple filesystems to create a distributed FS. I wouldn't use symlinks but indicate additional semantics via some metadata. > If you want to get creative along these lines without breaking Unix, I think creating some new type of artificial directory that doesn?t actually physically contain files but can flexibly reference a collection of files based on some sort of query - that might be interesting. Lot of such variations are possible using fusefs From wojtek at wojtek.tensor.gdynia.pl Thu Feb 13 09:15:04 2014 From: wojtek at wojtek.tensor.gdynia.pl (Wojciech Puchar) Date: Thu, 13 Feb 2014 10:14:45 +0100 (CET) Subject: FreeBSD UDF support In-Reply-To: <52FA94BF.80304@gwdg.de> References: <52FA94BF.80304@gwdg.de> Message-ID: > His project comes from NetBSD and modernises the UDF standard towards > 2.5x. With this driver, you would be able to read modern not encrypted > Blu-rays and DVDs. The driver works in principle, some minor issues have > to be solved. > > Unfortunately there is only very little interest to integrate this > driver into base. > i need RW support so it is not much use for me. From brueffer at FreeBSD.org Thu Feb 13 09:20:40 2014 From: brueffer at FreeBSD.org (Christian Brueffer) Date: Thu, 13 Feb 2014 10:20:33 +0100 Subject: FreeBSD UDF support In-Reply-To: References: <52FA94BF.80304@gwdg.de> Message-ID: <52FC8E61.5050200@FreeBSD.org> On 2/13/14 10:14 AM, Wojciech Puchar wrote: >> His project comes from NetBSD and modernises the UDF standard towards >> 2.5x. With this driver, you would be able to read modern not encrypted >> Blu-rays and DVDs. The driver works in principle, some minor issues have >> to be solved. >> >> Unfortunately there is only very little interest to integrate this >> driver into base. >> > i need RW support so it is not much use for me. NetBSD has newfs_udf(8), but it doesn't look like anyone has ported it to FreeBSD yet, Chris -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 1036 bytes Desc: OpenPGP digital signature URL: From vpenkoff at gmail.com Thu Feb 13 09:05:32 2014 From: vpenkoff at gmail.com (Viktor Penkoff) Date: Thu, 13 Feb 2014 11:05:31 +0200 Subject: Receiving jumbo frames Message-ID: Hi, folks! I'm writing an extension functionality to not-yet published network driver. I'm receiving the typical ethernet frames without problems. Considering the datasheet of the device, I'm capable of receiving jumbo frames. When I try to do that, e.g. to send jumbo frame of 8000 bytes, I'm receiving only a limited count of them - 105, then the kernel crashes with the following message: "panic: vm_fault: fault on nofault entry, addr: cfcec000". I have inspected a kernel dump with kgdb and the problem occurs at the function bus_dmamap_sync. The line that the panic happens is in a standard invoking receive interrupt function. Packet data is stored in memory buffers, with any single packet spanning multiple buffers if necessary. The buffers are allocated by the CPU and are managed through chained descriptor lists. When I'm sending jumbo packets, e.g. 8008 bytes in size each, due to live debugging with jtag and kgdb, I've got the following information for the mbuf: % ping -I eth0 -c 1 -s 8000 -M dont A.B.C.D -v -p ff {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc3d41800 "", mh_len = 2048, mh_flags = 3, mh_type = 1, pad = "\000"}, M_dat = { MH = {MH_pkthdr = {rcvif = 0x0, header = 0x0, len = 2048, flowid = 0, csum_flags = 0, csum_data = 0, tso_segsz = 0, PH_vt = { vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first = 0x0}, dsa_tag = {0, 0}}, MH_dat = {MH_ext = {ext_buf = 0xc3d41800 "", ext_free = 0, ext_arg1 = 0x0, ext_arg2 = 0x0, ext_size = 2048, ref_cnt = 0xc3d3f244, ext_type = 6}, In this case, the given mbuf must occupy ~4 descriptors, IMO. Following the next pointer in the descriptor list shows that the CPU has an ownership. Nevertheless, ?he byte count of the received frame shows that 8056 bytes are received. More interesting is the fact that an echo reply is given back. But after a constant count of received packets, the kernel hangs. Some background information: To enable the jumbo frame, one must set the appropriate register. At the software level, a ring buffer with the descriptors is implemented. The device, under which the driver runs, is arm-based. The freebsd version is 8.0. The device uses SerialDMA queues for transmitting and receiving. To receive packets, the CPU must perform the following: 1. Prepare a linked list of descriptors 2. Configure a given queue with the address of the first descriptor in the list, 3. enable SerialDMA; With the transmission - I don't have any problems. The logic is the same as by the reception of packets - ring buffer with descriptors. Any ideas what can cause this type of crashes? From ian at FreeBSD.org Thu Feb 13 14:50:25 2014 From: ian at FreeBSD.org (Ian Lepore) Date: Thu, 13 Feb 2014 07:50:12 -0700 Subject: Receiving jumbo frames In-Reply-To: References: Message-ID: <1392303012.1145.81.camel@revolution.hippie.lan> On Thu, 2014-02-13 at 11:05 +0200, Viktor Penkoff wrote: > Hi, folks! I'm writing an extension functionality to not-yet published > network driver. > I'm receiving the typical ethernet frames without problems. Considering the > datasheet of the device, > I'm capable of receiving jumbo frames. When I try to do that, e.g. to send > jumbo frame of 8000 bytes, I'm receiving only a limited count of them - > 105, then the kernel crashes with the following message: > "panic: vm_fault: fault on nofault entry, addr: cfcec000". > I have inspected a kernel dump with kgdb and the problem occurs at the > function bus_dmamap_sync. > > The line that the panic happens is in a standard invoking receive > interrupt function. Packet data is stored in memory buffers, with any > single packet spanning multiple buffers if necessary. The buffers are > allocated by the CPU and are managed through chained descriptor lists. > When I'm sending jumbo packets, e.g. 8008 bytes in size each, due to live > debugging with jtag and kgdb, I've got the following information for the > mbuf: > > % ping -I eth0 -c 1 -s 8000 -M dont A.B.C.D -v -p ff > > {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc3d41800 "", > mh_len = 2048, mh_flags = 3, mh_type = 1, pad = "\000"}, M_dat = { > MH = {MH_pkthdr = {rcvif = 0x0, header = 0x0, len = 2048, flowid = 0, > csum_flags = 0, csum_data = 0, tso_segsz = 0, PH_vt = { > vt_vtag = 0, vt_nrecs = 0}, tags = {slh_first = 0x0}, > dsa_tag = {0, 0}}, MH_dat = {MH_ext = {ext_buf = 0xc3d41800 "", > ext_free = 0, ext_arg1 = 0x0, ext_arg2 = 0x0, ext_size = 2048, > ref_cnt = 0xc3d3f244, ext_type = 6}, > > In this case, the given mbuf must occupy ~4 descriptors, IMO. Following the > next pointer in the descriptor list shows that the CPU has > an ownership. Nevertheless, ?he byte count of the received frame shows that > 8056 bytes are received. > > More interesting is the fact that an echo reply is given back. But after a > constant count of received packets, the kernel hangs. > > Some background information: > To enable the jumbo frame, one must set the appropriate register. > At the software level, a ring buffer with the descriptors is implemented. > The device, under which the driver runs, is arm-based. > The freebsd version is 8.0. The device uses SerialDMA queues for > transmitting and receiving. > To receive packets, the CPU must perform the following: > 1. Prepare a linked list of descriptors > 2. Configure a given queue with the address of the first descriptor in > the list, > 3. enable SerialDMA; > > With the transmission - I don't have any problems. The logic is the same as > by the reception of packets - ring buffer with descriptors. > > Any ideas what can cause this type of crashes? You mention FreeBSD 8.0, that's pretty old. There have been many fixes to the arm busdma code since then. We still use 8.2 at work, and while we know there are problems with the busdma code that have been fixed in later revisions, they aren't affecting us so we leave well enough alone. We don't use jumbo frames though. At the very least you should apply the changeset r236086 to your code, although it's not all that likely to fix your problem because the mbufs you're working with should be aligned on 2k boundaries and never require a partial cacheline flush. Are you allocating the buffers with bus_dmamem_alloc() then attaching them to mbuf chains as external buffers? Or are you using one of the mbuf allocation routines? When interfacing with the busdma code, are you using bus_dmamap_load_mbuf()? Posting the full backtrace from the panic might help. More information about the arm chip/system you're using might be helpful too. (I've added the freebsd-arm mailing list to the CC, this is pretty likely to be an arm-specific problem rather than a general freebsd thing.) -- Ian From wblock at wonkity.com Thu Feb 13 15:30:35 2014 From: wblock at wonkity.com (Warren Block) Date: Thu, 13 Feb 2014 08:30:33 -0700 (MST) Subject: FreeBSD UDF support In-Reply-To: <52FC8E61.5050200@FreeBSD.org> References: <52FA94BF.80304@gwdg.de> <52FC8E61.5050200@FreeBSD.org> Message-ID: On Thu, 13 Feb 2014, Christian Brueffer wrote: > On 2/13/14 10:14 AM, Wojciech Puchar wrote: >>> His project comes from NetBSD and modernises the UDF standard towards >>> 2.5x. With this driver, you would be able to read modern not encrypted >>> Blu-rays and DVDs. The driver works in principle, some minor issues have >>> to be solved. >>> >>> Unfortunately there is only very little interest to integrate this >>> driver into base. >>> >> i need RW support so it is not much use for me. > > NetBSD has newfs_udf(8), but it doesn't look like anyone has ported it > to FreeBSD yet, GSOC candidates have been requested. That might be perfect. From vijju.singh at gmail.com Thu Feb 13 20:51:51 2014 From: vijju.singh at gmail.com (Vijay Singh) Date: Thu, 13 Feb 2014 12:51:50 -0800 Subject: Debugging rw lock Message-ID: I am running into an issue where an rw lock is read locked and never unlocked, and causes a system to livelock. I was wondering if its possible to figure out which thread owns the read lock? It's the tcp pcbinfo lock. (kgdb-amd64-7.4-08) show_rwlock rw name : tcp class: rw flags: {SLEEP, INITED, WITNESS, RECURSE, UPGRADABLE} state: RLOCK: 1 locks waiters: writers Any help is appreciated. -vijay From bright at mu.org Thu Feb 13 20:53:35 2014 From: bright at mu.org (Alfred Perlstein) Date: Thu, 13 Feb 2014 12:53:45 -0800 Subject: Debugging rw lock In-Reply-To: References: Message-ID: <52FD30D9.6050604@mu.org> Keep a stack of rwlocks owned in the struct thread. -Alfred On 2/13/14, 12:51 PM, Vijay Singh wrote: > I am running into an issue where an rw lock is read locked and never > unlocked, and causes a system to livelock. I was wondering if its possible > to figure out which thread owns the read lock? > > It's the tcp pcbinfo lock. > > (kgdb-amd64-7.4-08) show_rwlock rw > name : tcp > class: rw > flags: {SLEEP, INITED, WITNESS, RECURSE, UPGRADABLE} > state: RLOCK: 1 locks > waiters: writers > > Any help is appreciated. > > -vijay > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" > From vijju.singh at gmail.com Thu Feb 13 20:59:11 2014 From: vijju.singh at gmail.com (Vijay Singh) Date: Thu, 13 Feb 2014 12:59:10 -0800 Subject: Debugging rw lock In-Reply-To: <52FD30D9.6050604@mu.org> References: <52FD30D9.6050604@mu.org> Message-ID: You're talking about instrumenting the code, right? But which thread? I was thinking of augmenting the rw lock to record the readers, but wanted to check if something is possible without instrumentation. On Thu, Feb 13, 2014 at 12:53 PM, Alfred Perlstein wrote: > Keep a stack of rwlocks owned in the struct thread. > > -Alfred > > On 2/13/14, 12:51 PM, Vijay Singh wrote: > >> I am running into an issue where an rw lock is read locked and never >> unlocked, and causes a system to livelock. I was wondering if its possible >> to figure out which thread owns the read lock? >> >> It's the tcp pcbinfo lock. >> >> (kgdb-amd64-7.4-08) show_rwlock rw >> name : tcp >> class: rw >> flags: {SLEEP, INITED, WITNESS, RECURSE, UPGRADABLE} >> state: RLOCK: 1 locks >> waiters: writers >> >> Any help is appreciated. >> >> -vijay >> _______________________________________________ >> freebsd-hackers at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org >> " >> >> > From bright at mu.org Thu Feb 13 21:02:42 2014 From: bright at mu.org (Alfred Perlstein) Date: Thu, 13 Feb 2014 13:02:55 -0800 Subject: Debugging rw lock In-Reply-To: References: <52FD30D9.6050604@mu.org> Message-ID: <52FD32FF.70106@mu.org> On 2/13/14, 12:59 PM, Vijay Singh wrote: > You're talking about instrumenting the code, right? But which thread? I was > thinking of augmenting the rw lock to record the readers, but wanted to > check if something is possible without instrumentation. If rw locks are implemented using low level atomics then you're going to make the very slow and have a LOT of work to do as opposed to just using a thread specific storage to implement it. You're better off just making use of the fact that only "curthread" can access the per-thread stack and just use that. Or at least that's how it works in my brain. -Alfred > > > On Thu, Feb 13, 2014 at 12:53 PM, Alfred Perlstein wrote: > >> Keep a stack of rwlocks owned in the struct thread. >> >> -Alfred >> >> On 2/13/14, 12:51 PM, Vijay Singh wrote: >> >>> I am running into an issue where an rw lock is read locked and never >>> unlocked, and causes a system to livelock. I was wondering if its possible >>> to figure out which thread owns the read lock? >>> >>> It's the tcp pcbinfo lock. >>> >>> (kgdb-amd64-7.4-08) show_rwlock rw >>> name : tcp >>> class: rw >>> flags: {SLEEP, INITED, WITNESS, RECURSE, UPGRADABLE} >>> state: RLOCK: 1 locks >>> waiters: writers >>> >>> Any help is appreciated. >>> >>> -vijay >>> _______________________________________________ >>> freebsd-hackers at freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org >>> " >>> >>> > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" > From ian at FreeBSD.org Thu Feb 13 21:06:33 2014 From: ian at FreeBSD.org (Ian Lepore) Date: Thu, 13 Feb 2014 14:06:27 -0700 Subject: Debugging rw lock In-Reply-To: References: <52FD30D9.6050604@mu.org> Message-ID: <1392325587.1145.96.camel@revolution.hippie.lan> Does option DEADLKRES not work with rwlocks? (I've never used it, just seen it in the NOTES). -- Ian On Thu, 2014-02-13 at 12:59 -0800, Vijay Singh wrote: > You're talking about instrumenting the code, right? But which thread? I was > thinking of augmenting the rw lock to record the readers, but wanted to > check if something is possible without instrumentation. > > > On Thu, Feb 13, 2014 at 12:53 PM, Alfred Perlstein wrote: > > > Keep a stack of rwlocks owned in the struct thread. > > > > -Alfred > > > > On 2/13/14, 12:51 PM, Vijay Singh wrote: > > > >> I am running into an issue where an rw lock is read locked and never > >> unlocked, and causes a system to livelock. I was wondering if its possible > >> to figure out which thread owns the read lock? > >> > >> It's the tcp pcbinfo lock. > >> > >> (kgdb-amd64-7.4-08) show_rwlock rw > >> name : tcp > >> class: rw > >> flags: {SLEEP, INITED, WITNESS, RECURSE, UPGRADABLE} > >> state: RLOCK: 1 locks > >> waiters: writers > >> > >> Any help is appreciated. > >> > >> -vijay > >> _______________________________________________ > >> freebsd-hackers at freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org > >> " > >> > >> > > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" From rysto32 at gmail.com Thu Feb 13 22:03:07 2014 From: rysto32 at gmail.com (Ryan Stone) Date: Thu, 13 Feb 2014 17:03:06 -0500 Subject: Debugging rw lock In-Reply-To: <1392325587.1145.96.camel@revolution.hippie.lan> References: <52FD30D9.6050604@mu.org> <1392325587.1145.96.camel@revolution.hippie.lan> Message-ID: On Thu, Feb 13, 2014 at 4:06 PM, Ian Lepore wrote: > Does option DEADLKRES not work with rwlocks? (I've never used it, just > seen it in the NOTES). > > -- Ian DEADLKRES will panic the system after a thread has been blocked for a timeout period (I think that it's something ridiculous like 30 minutes). However it's of no use when trying to debugging a leaked read lock, because the thread that lost the lock will have left behind no clues as to it's identity. If you have some kind of reproduction scenario, the quickest hack that might get you an answer would be to change INP_INFO_RLOCK to actually take a wlock instead. Rather than instrumenting the code, you could use the dtrace lockstat provider to log every lock/unlock and post-process the output to find the culprit. Make sure that you have r258541 if you want to use lockstat. http://svnweb.freebsd.org/changeset/base/258541 From adrian at freebsd.org Thu Feb 13 23:57:34 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Thu, 13 Feb 2014 15:57:33 -0800 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? Message-ID: Hi, Whilst digging into collisions in the flowtable code I discovered that a bunch of them are due to some of the flowtable_lookup() code running on a different CPU - the contention on the l2/l3 lookup lock(s) was enough to block things so they'd get an obvious chance to be migrated. So this led me to wonder whether in a fully preemptive kernel, a running kernel thread would stay on the current CPU until it hit a very specific subset of things (exited to userland, hit a lock, etc.) Apparently (according to kan and rwatson) this is how we define fully preemptive - it's not just interruptable at almost any point, but the running task may be interrupted and rescheduled on a different CPU outside of specific critical sections. This means that for the flowtable case, the current setup (without atomics to maintain the lists) can only work if all threads doing work with the flowtable structures (ie, lookup, insert, purge) have to be CPU pinned. Otherwise we may have the situation where: sequentually: * lookup occurs on CPU A; * lookup succeeds on CPU A for some almost-expired entry; * preemption occurs, and it gets scheduled to CPU B; then simultaneously: * CPU A's flowtable purge code runs, and decides to purge entries including the current one; * the code now running on CPU B has an item from the CPU A flowtable, and dereferences it as it's being freed, leading to potential badness. Now, it's a ridiculously small window of opportunity, but I'd rather the code be written to be correct and mostly-fast versus very fast and potentially exploding. I'm sure those in operations would agree. :-) So, my questions: * is this actually how fully pre-emptive kernels _may_ behave? * I believe there's a difference between what 4BSD and ULE will do here - is this the case? What are the scheduler behaviours? * are there any other areas in the kernel that rely on pcpu uma zones / curcpu indexes for things outside of sched_pin() ? Thanks, -a From Suresh.Gumpula at netapp.com Fri Feb 14 02:07:55 2014 From: Suresh.Gumpula at netapp.com (Gumpula, Suresh) Date: Fri, 14 Feb 2014 02:07:51 +0000 Subject: malloc(9) and its alignment In-Reply-To: <20140212220705.GV34851@funkthat.com> References: <1392214788.1145.52.camel@revolution.hippie.lan> <20140212220705.GV34851@funkthat.com> Message-ID: Thanks for the explanation John. How about porting ARM way of handling required alignment with creation of separate zones And allocating with uma_zalloc(zone) for AMD64 too for bus_dmamem_alloc? Thanks Suresh -----Original Message----- From: John-Mark Gurney [mailto:jmg at funkthat.com] Sent: Wednesday, February 12, 2014 5:07 PM To: Gumpula, Suresh Cc: Ian Lepore; freebsd-hackers at freebsd.org Subject: Re: malloc(9) and its alignment Gumpula, Suresh wrote this message on Wed, Feb 12, 2014 at 19:40 +0000: > Thanks Ian for the reply. I will look at the ARM code, but I was thinking why malloc(9) does not return bucket size aligned pointers. Always returning bucket sizes aligned pointers may not be ideal for a cache.. say you have a buffer of 512 bytes, where often only the first 128 bytes are used (but all 512 bytes may be)... If you always align at 512, some cache lines will be more heavily used than others, reducing the effective size of the cache... This is only one reason not aligning to size may be better... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From matthias.andree at gmx.de Fri Feb 14 02:14:14 2014 From: matthias.andree at gmx.de (Matthias Andree) Date: Fri, 14 Feb 2014 03:14:04 +0100 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: References: Message-ID: <52FD7BEC.9050807@gmx.de> Am 14.02.2014 00:57, schrieb Adrian Chadd: > Hi, > > Whilst digging into collisions in the flowtable code I discovered that > a bunch of them are due to some of the flowtable_lookup() code running > on a different CPU - the contention on the l2/l3 lookup lock(s) was > enough to block things so they'd get an obvious chance to be migrated. > > So this led me to wonder whether in a fully preemptive kernel, a > running kernel thread would stay on the current CPU until it hit a > very specific subset of things (exited to userland, hit a lock, etc.) > > Apparently (according to kan and rwatson) this is how we define fully > preemptive - it's not just interruptable at almost any point, but the > running task may be interrupted and rescheduled on a different CPU > outside of specific critical sections. There's one more question to ask, and that is if you have a thread that warmed up its caches and you migrate it to a different core - what happens with the execution time? It might become longer because you don't usually share or migrate the L1/L2 cache contents, so I'd naively expect lower cache hit ratios. Certainly there is not a simple answer to that, but should someone start thinking about scheduler implementations, the "setup overhead" for the switch to a different core might be relevant. (Whether it's really more than the extra effort a scheduler would require to spend to make a better decision is then yet one more question.) From rysto32 at gmail.com Fri Feb 14 02:37:05 2014 From: rysto32 at gmail.com (Ryan Stone) Date: Thu, 13 Feb 2014 21:37:04 -0500 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: References: Message-ID: On Thu, Feb 13, 2014 at 6:57 PM, Adrian Chadd wrote: > sequentually: > > * lookup occurs on CPU A; > * lookup succeeds on CPU A for some almost-expired entry; > * preemption occurs, and it gets scheduled to CPU B; > > then simultaneously: > > * CPU A's flowtable purge code runs, and decides to purge entries > including the current one; > * the code now running on CPU B has an item from the CPU A flowtable, > and dereferences it as it's being freed, leading to potential badness. This kind of scenario is definitely possible. All of the FreeBSD kernel code that deals with lockless per-cpu data structures that I have seen (e.g. uma) has used critical_enter()/critical_exit() to prevent preemption, and have been careful to invalidate their references to the per-cpu data if they have to drop the critical section. I don't believe that sched_pin() is good enough because I don't believe that it handles the scenario when thread A gets a reference and then is preempted, thread B frees the entry, and then A is scheduled and uses the now-freed entry. However I'm really not familiar at all with flowtable so maybe there's something preventing that. From adrian at freebsd.org Fri Feb 14 03:06:09 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Thu, 13 Feb 2014 19:06:07 -0800 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: References: Message-ID: On 13 February 2014 18:37, Ryan Stone wrote: > On Thu, Feb 13, 2014 at 6:57 PM, Adrian Chadd wrote: >> sequentually: >> >> * lookup occurs on CPU A; >> * lookup succeeds on CPU A for some almost-expired entry; >> * preemption occurs, and it gets scheduled to CPU B; >> >> then simultaneously: >> >> * CPU A's flowtable purge code runs, and decides to purge entries >> including the current one; >> * the code now running on CPU B has an item from the CPU A flowtable, >> and dereferences it as it's being freed, leading to potential badness. > > This kind of scenario is definitely possible. All of the FreeBSD > kernel code that deals with lockless per-cpu data structures that I > have seen (e.g. uma) has used critical_enter()/critical_exit() to > prevent preemption, and have been careful to invalidate their > references to the per-cpu data if they have to drop the critical > section. > > I don't believe that sched_pin() is good enough because I don't > believe that it handles the scenario when thread A gets a reference > and then is preempted, thread B frees the entry, and then A is > scheduled and uses the now-freed entry. However I'm really not > familiar at all with flowtable so maybe there's something preventing > that. Oh man, I hate you for bringing that up. So, both flowtable_lookup_common() and flowtable_insert() does do a critical_enter() / critical_exit() during the list operations, so that's ok. It won't be pre-empted. If I wrap flowtable_lookup() with sched_pin() so it covers both the lookup path and the insert path, then it won't get scheduled on another CPU. Then the critical sections around the list operations guarantee it won't end up with a freed entry. The fl entry uptime is updated in the critical section so it shouldn't end up overlapping with the flow purger in a way that has it removed. So, I think that's okay. I think once i add sched_pin() in the right spots, this crap won't use a stale entry. Well, as long as the entry get used before the flow cleaner picks it up. Phew! -adrian From Mark.Martinec+freebsd at ijs.si Fri Feb 14 01:40:14 2014 From: Mark.Martinec+freebsd at ijs.si (Mark Martinec) Date: Fri, 14 Feb 2014 02:40:07 +0100 Subject: Merge ping+ping6 and traceroue+traceroute6 to single =?UTF-8?Q?utilities=3F?= In-Reply-To: References: <1063008459.20140111160525@serebryakov.spb.ru> <52D14140.3090003@gibfest.dk> <20140111.143644.41639035.sthaug@nethelp.no> <52D15185.50802@gibfest.dk> <0c45208d87526a80f559ac09e28163c2.authenticated@ultimatedns.net> Message-ID: <3feb1ef62b087b8bf00f34c42c5c2954@mailbox.ijs.si> 2014-02-11, Mark Martinec wrote: > Remember the original PHK's story ( http://bikeshed.com/ ) ? > It ended favourably for the sleep(1) command, it got its new feature. > What can be learned there is: just needs someone to do it and be > persistent enough to be accepted. > > Looks like a perfect task for Google Summer of Code 2014, > time to apply is very near: > http://www.google-melange.com/gsoc/homepage/google/gsoc2014 2014-02-12, Kevin Oberman wrote: > For those who are new at IPv6, the ping6 and traceroute6 commands come > from > the WIDE KAME project. KAME developed one of the earliest IPv6 stacks > and > WIDE used FreeBSD. It became the FreeBSD IPv6 stack and the ping6 and > traceroute6 utilities were brought in with the rest of the KAME code. > > When these tools were written, the IPv6 stack and the supporting > libraries > and APIs were very primitive. I suspect that it was quicker to write > new > tools than to try to integrate IPv6 into the existing standard tools > and, > when things were so rough, there was a clear effort to avoid changes to > working IPv4 code. Separate IPv4 and IPv6 tools made sense then, but > the > need has long vanished... probably even before the KAME project ended. > But > the old, separate tools lived on through simple inertia. > > And so it is today. Inertia is NO reason that it should be this way > forever. I have submitted two entries for FreeBSD Google Summer of Code 2014: https://wiki.freebsd.org/SummerOfCode2014 (should show up there eventually after a review, I hope), one for a unified ping and ping6, the other for a unified traceroute and traceroute6. My first impression was that it may be possible to do both in a single 12 week GSoC job, although after checking existing source code and writing the proposal it now looks to me more like two full-time summer jobs, if they are to be done properly and with attention to details. Looking for one, or preferably two, mentors for students for these tasks. I wonder if Bjoern A. Zeeb wouldn't be the best man for the job ;) Mark From jhellenthal at dataix.net Fri Feb 14 03:24:39 2014 From: jhellenthal at dataix.net (Jason Hellenthal) Date: Thu, 13 Feb 2014 22:24:33 -0500 Subject: Merge ping+ping6 and traceroue+traceroute6 to single utilities? In-Reply-To: <3feb1ef62b087b8bf00f34c42c5c2954@mailbox.ijs.si> References: <1063008459.20140111160525@serebryakov.spb.ru> <52D14140.3090003@gibfest.dk> <20140111.143644.41639035.sthaug@nethelp.no> <52D15185.50802@gibfest.dk> <0c45208d87526a80f559ac09e28163c2.authenticated@ultimatedns.net> <3feb1ef62b087b8bf00f34c42c5c2954@mailbox.ijs.si> Message-ID: <05380FA6-4E24-4BFA-AB96-81665C867ABA@dataix.net> > On Feb 13, 2014, at 20:40, Mark Martinec wrote: > > 2014-02-11, Mark Martinec wrote: >> Remember the original PHK's story ( http://bikeshed.com/ ) ? >> It ended favourably for the sleep(1) command, it got its new feature. >> What can be learned there is: just needs someone to do it and be >> persistent enough to be accepted. >> Looks like a perfect task for Google Summer of Code 2014, >> time to apply is very near: >> http://www.google-melange.com/gsoc/homepage/google/gsoc2014 > > > 2014-02-12, Kevin Oberman wrote: >> For those who are new at IPv6, the ping6 and traceroute6 commands come from >> the WIDE KAME project. KAME developed one of the earliest IPv6 stacks and >> WIDE used FreeBSD. It became the FreeBSD IPv6 stack and the ping6 and >> traceroute6 utilities were brought in with the rest of the KAME code. >> When these tools were written, the IPv6 stack and the supporting libraries >> and APIs were very primitive. I suspect that it was quicker to write new >> tools than to try to integrate IPv6 into the existing standard tools and, >> when things were so rough, there was a clear effort to avoid changes to >> working IPv4 code. Separate IPv4 and IPv6 tools made sense then, but the >> need has long vanished... probably even before the KAME project ended. But >> the old, separate tools lived on through simple inertia. >> And so it is today. Inertia is NO reason that it should be this way forever. > > > I have submitted two entries for FreeBSD Google Summer of Code 2014: > > https://wiki.freebsd.org/SummerOfCode2014 > > (should show up there eventually after a review, I hope), > > one for a unified ping and ping6, the other for a unified traceroute > and traceroute6. My first impression was that it may be possible to do > both in a single 12 week GSoC job, although after checking existing > source code and writing the proposal it now looks to me more like > two full-time summer jobs, if they are to be done properly and with > attention to details. > > Looking for one, or preferably two, mentors for students for these tasks. > I wonder if Bjoern A. Zeeb wouldn't be the best man for the job ;) > Awesome, personally that would seem like the best route not only to have the focus on the tool itself but to put focus on achieving one or another either way it's progress. If we were voting I couldn't agree more with Bjoern if he would accept it.. Couldn't imagine someone more in depth to fit the task. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6118 bytes Desc: not available URL: From julian at freebsd.org Fri Feb 14 03:48:03 2014 From: julian at freebsd.org (Julian Elischer) Date: Fri, 14 Feb 2014 11:47:46 +0800 Subject: Debugging rw lock In-Reply-To: <52FD32FF.70106@mu.org> References: <52FD30D9.6050604@mu.org> <52FD32FF.70106@mu.org> Message-ID: <52FD91E2.4080903@freebsd.org> On 2/14/14, 5:02 AM, Alfred Perlstein wrote: > > On 2/13/14, 12:59 PM, Vijay Singh wrote: >> You're talking about instrumenting the code, right? But which >> thread? I was >> thinking of augmenting the rw lock to record the readers, but >> wanted to >> check if something is possible without instrumentation. > > If rw locks are implemented using low level atomics then you're > going to make the very slow and have a LOT of work to do as opposed > to just using a thread specific storage to implement it. You're > better off just making use of the fact that only "curthread" can > access the per-thread stack and just use that. Or at least that's > how it works in my brain. maybe do it the other way around and count the read locks a thread has.. then when there is a problem go see what they hold.. (or add the ability to detail 10 locks each or something) > > -Alfred > > >> >> >> On Thu, Feb 13, 2014 at 12:53 PM, Alfred Perlstein >> wrote: >> >>> Keep a stack of rwlocks owned in the struct thread. >>> >>> -Alfred >>> >>> On 2/13/14, 12:51 PM, Vijay Singh wrote: >>> >>>> I am running into an issue where an rw lock is read locked and never >>>> unlocked, and causes a system to livelock. I was wondering if its >>>> possible >>>> to figure out which thread owns the read lock? >>>> >>>> It's the tcp pcbinfo lock. >>>> >>>> (kgdb-amd64-7.4-08) show_rwlock rw >>>> name : tcp >>>> class: rw >>>> flags: {SLEEP, INITED, WITNESS, RECURSE, UPGRADABLE} >>>> state: RLOCK: 1 locks >>>> waiters: writers >>>> >>>> Any help is appreciated. >>>> >>>> -vijay >>>> _______________________________________________ >>>> freebsd-hackers at freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>>> To unsubscribe, send any mail to >>>> "freebsd-hackers-unsubscribe at freebsd.org >>>> " >>>> >>>> >> _______________________________________________ >> freebsd-hackers at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to >> "freebsd-hackers-unsubscribe at freebsd.org" >> > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe at freebsd.org" > From jim at netgate.com Fri Feb 14 05:38:03 2014 From: jim at netgate.com (Jim Thompson) Date: Thu, 13 Feb 2014 23:37:49 -0600 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: Message-ID: <40637D5C-EBBB-49C0-BE82-BA644C32D778@netgate.com> On Feb 12, 2014, at 11:17 PM, Jordan Hubbard wrote: > > On Feb 12, 2014, at 9:08 PM, Jordan Hubbard wrote: > >> Globbing is done in user land (by the shell) - you wouldn?t want to push that down into the kernel, which is either what you?d have to do or you?d need a user land daemon which did round-trips with the kernel to do the translation, which would also need to make sure to get all of the process permission stuff right since the user id / gid / $CWD would all potentially affect the expansion of the ?symlink?. > > Actually, just to correct myself, there is a third way, which is that you could make the shell also do the expansion of the symlink (or interpose it into libc), but now you?d just be stacking one weird hack on top of another weird hack. It?s still not a good idea for all the reasons I mentioned, at least not as a ?symlink?. Maybe some new type of shell builtin, though I?m not sure how/where you?d use it. There is a fourth way. Just embed a (call to a) shell script in the symlink. If Pyramid?s conditional symbolic links were the computed goto of the filesystem, and variant symlinks are a modern .. er? variant of same, then embedding a call to an external program has to be the Turing-complete ? er? variant. amiright? What could go wrong? :-) OK, seriously.. once upon a time there was an OS named Mach. It had the concept of a name server. Pathname resolution worked like this: consider /mnt/readme.txt where /mnt is a mounted filesystem. ? libc asks the root filesystem server about /mnt/readme.txt. ? The root filesystem returns a port to the mnt filesystem server (matching /mnt) and the retry name /readme.txt. ? libc asks the mnt filesystem server about /readme.txt. ? The mnt filesystem server returns a port to itself and records that this port refers to the regular file /readme.txt. While a regular filesystem server will just serve the data as stored in a filesystem on disk, on Mach there can be servers providing purely virtual information, or a mixture of both. In Mach-land, it is up to the server to behave and provide consistent and useful data on each remote procedure call. If it does not, the results may not match the expectations of the user and confuse him. You are lost in a maze of twisty little filesystems, all alike?. OK, more seriously. This seems straight-forward to implement via FUSE. You might look at fsfipl & filterFS http://sourceforge.net/projects/fsfipi/ http://filterfs.sourceforge.net. From adrian at freebsd.org Fri Feb 14 09:22:35 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Fri, 14 Feb 2014 01:22:34 -0800 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: References: Message-ID: Ok, so now I remember the other odd thing. I was seeing the sending context(s) jumping from one CPU to another during flowtable_insert_common(), around the locking bits. But I thread pinned all the sender user threads! So, why would the senders still be scheduled on other CPUs if I've pinned the userland threads? (and yes, I verified that the userland threads weren't moving around.) -a From jhb at freebsd.org Fri Feb 14 17:51:34 2014 From: jhb at freebsd.org (John Baldwin) Date: Fri, 14 Feb 2014 11:39:48 -0500 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: References: Message-ID: <201402141139.49158.jhb@freebsd.org> On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: > Ok, so now I remember the other odd thing. > > I was seeing the sending context(s) jumping from one CPU to another > during flowtable_insert_common(), around the locking bits. > > But I thread pinned all the sender user threads! > > So, why would the senders still be scheduled on other CPUs if I've > pinned the userland threads? > > (and yes, I verified that the userland threads weren't moving around.) Can you clarify a bit? It's not clear how sender thraeds differ from userland threads differ from sender user threads. (I.e. one reading is that these are all the same thing and should thus all be pinned (I assume you mean using cpuset to bind them to specific cores rather than sched_pin)) -- John Baldwin From adrian at freebsd.org Fri Feb 14 17:55:05 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Fri, 14 Feb 2014 09:55:03 -0800 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <201402141139.49158.jhb@freebsd.org> References: <201402141139.49158.jhb@freebsd.org> Message-ID: On 14 February 2014 08:39, John Baldwin wrote: > On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: >> Ok, so now I remember the other odd thing. >> >> I was seeing the sending context(s) jumping from one CPU to another >> during flowtable_insert_common(), around the locking bits. >> >> But I thread pinned all the sender user threads! >> >> So, why would the senders still be scheduled on other CPUs if I've >> pinned the userland threads? >> >> (and yes, I verified that the userland threads weren't moving around.) > > Can you clarify a bit? It's not clear how sender thraeds differ from > userland threads differ from sender user threads. (I.e. one reading > is that these are all the same thing and should thus all be pinned > (I assume you mean using cpuset to bind them to specific cores rather > than sched_pin)) Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: * the userland threads are using the cpuset call to map a thread into a cpuset, yes * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as the userland threads -a From jhb at freebsd.org Fri Feb 14 18:18:51 2014 From: jhb at freebsd.org (John Baldwin) Date: Fri, 14 Feb 2014 12:59:15 -0500 Subject: Debugging rw lock In-Reply-To: <52FD91E2.4080903@freebsd.org> References: <52FD32FF.70106@mu.org> <52FD91E2.4080903@freebsd.org> Message-ID: <201402141259.15712.jhb@freebsd.org> On Thursday, February 13, 2014 10:47:46 pm Julian Elischer wrote: > On 2/14/14, 5:02 AM, Alfred Perlstein wrote: > > > > On 2/13/14, 12:59 PM, Vijay Singh wrote: > >> You're talking about instrumenting the code, right? But which > >> thread? I was > >> thinking of augmenting the rw lock to record the readers, but > >> wanted to > >> check if something is possible without instrumentation. > > > > If rw locks are implemented using low level atomics then you're > > going to make the very slow and have a LOT of work to do as opposed > > to just using a thread specific storage to implement it. You're > > better off just making use of the fact that only "curthread" can > > access the per-thread stack and just use that. Or at least that's > > how it works in my brain. > > maybe do it the other way around and count the read locks a thread > has.. then when there is a problem go see what they hold.. > (or add the ability to detail 10 locks each or something) We already do that. Vijay, you can look at td_rw_rlocks in each struct thread. You can at least ignore the threads with a count set to zero. The other option is to run with WITNESS enabled. You can then do 'show all locks' in ddb and you can see which thread owns the lock (and the file and line number where it acquired it) -- John Baldwin From jhb at freebsd.org Fri Feb 14 18:18:54 2014 From: jhb at freebsd.org (John Baldwin) Date: Fri, 14 Feb 2014 13:18:44 -0500 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: References: <201402141139.49158.jhb@freebsd.org> Message-ID: <201402141318.44743.jhb@freebsd.org> On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote: > On 14 February 2014 08:39, John Baldwin wrote: > > On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: > >> Ok, so now I remember the other odd thing. > >> > >> I was seeing the sending context(s) jumping from one CPU to another > >> during flowtable_insert_common(), around the locking bits. > >> > >> But I thread pinned all the sender user threads! > >> > >> So, why would the senders still be scheduled on other CPUs if I've > >> pinned the userland threads? > >> > >> (and yes, I verified that the userland threads weren't moving around.) > > > > Can you clarify a bit? It's not clear how sender thraeds differ from > > userland threads differ from sender user threads. (I.e. one reading > > is that these are all the same thing and should thus all be pinned > > (I assume you mean using cpuset to bind them to specific cores rather > > than sched_pin)) > > Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: > > * the userland threads are using the cpuset call to map a thread into > a cpuset, yes > * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as > the userland threads If they are all cpuset to a single CPU, they should not migrate, though I think sched_bind() can override that. However, that requires code to explicitly call sched_bind() which should be rare. -- John Baldwin From ache at freebsd.org Fri Feb 14 18:26:16 2014 From: ache at freebsd.org (Andrey Chernov) Date: Fri, 14 Feb 2014 22:26:07 +0400 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <201402141318.44743.jhb@freebsd.org> References: <201402141139.49158.jhb@freebsd.org> <201402141318.44743.jhb@freebsd.org> Message-ID: <52FE5FBF.3090104@freebsd.org> On 14.02.2014 22:18, John Baldwin wrote: > On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote: >> On 14 February 2014 08:39, John Baldwin wrote: >>> On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: >>>> Ok, so now I remember the other odd thing. >>>> >>>> I was seeing the sending context(s) jumping from one CPU to another >>>> during flowtable_insert_common(), around the locking bits. >>>> >>>> But I thread pinned all the sender user threads! >>>> >>>> So, why would the senders still be scheduled on other CPUs if I've >>>> pinned the userland threads? >>>> >>>> (and yes, I verified that the userland threads weren't moving around.) >>> >>> Can you clarify a bit? It's not clear how sender thraeds differ from >>> userland threads differ from sender user threads. (I.e. one reading >>> is that these are all the same thing and should thus all be pinned >>> (I assume you mean using cpuset to bind them to specific cores rather >>> than sched_pin)) >> >> Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: >> >> * the userland threads are using the cpuset call to map a thread into >> a cpuset, yes >> * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as >> the userland threads > > If they are all cpuset to a single CPU, they should not migrate, though > I think sched_bind() can override that. However, that requires code to > explicitly call sched_bind() which should be rare. > Due to this bug, not fixed yet, the real picture is more complex: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 -- http://ache.vniz.net/ From jhb at freebsd.org Fri Feb 14 19:10:44 2014 From: jhb at freebsd.org (John Baldwin) Date: Fri, 14 Feb 2014 14:10:29 -0500 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <52FE5FBF.3090104@freebsd.org> References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> Message-ID: <201402141410.29325.jhb@freebsd.org> On Friday, February 14, 2014 1:26:07 pm Andrey Chernov wrote: > On 14.02.2014 22:18, John Baldwin wrote: > > On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote: > >> On 14 February 2014 08:39, John Baldwin wrote: > >>> On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: > >>>> Ok, so now I remember the other odd thing. > >>>> > >>>> I was seeing the sending context(s) jumping from one CPU to another > >>>> during flowtable_insert_common(), around the locking bits. > >>>> > >>>> But I thread pinned all the sender user threads! > >>>> > >>>> So, why would the senders still be scheduled on other CPUs if I've > >>>> pinned the userland threads? > >>>> > >>>> (and yes, I verified that the userland threads weren't moving around.) > >>> > >>> Can you clarify a bit? It's not clear how sender thraeds differ from > >>> userland threads differ from sender user threads. (I.e. one reading > >>> is that these are all the same thing and should thus all be pinned > >>> (I assume you mean using cpuset to bind them to specific cores rather > >>> than sched_pin)) > >> > >> Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: > >> > >> * the userland threads are using the cpuset call to map a thread into > >> a cpuset, yes > >> * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as > >> the userland threads > > > > If they are all cpuset to a single CPU, they should not migrate, though > > I think sched_bind() can override that. However, that requires code to > > explicitly call sched_bind() which should be rare. > > > > Due to this bug, not fixed yet, the real picture is more complex: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 Eh, that bug report has no useful details, as in, it doesn't list the actual commands run. If you do 'cpuset -l 6 -s 1' to force all processes to only use CPU6, then yes, of course the other CPUs are idle because that's what you _asked_ for. AFAICT, that is all the original reporter did. At work we regularly add and remove CPUs from the default set (set 1) on hundreds of machines every day with ULE without any issues. -- John Baldwin From adrian at freebsd.org Fri Feb 14 19:18:31 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Fri, 14 Feb 2014 11:18:30 -0800 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <201402141318.44743.jhb@freebsd.org> References: <201402141139.49158.jhb@freebsd.org> <201402141318.44743.jhb@freebsd.org> Message-ID: On 14 February 2014 10:18, John Baldwin wrote: > If they are all cpuset to a single CPU, they should not migrate, though > I think sched_bind() can override that. However, that requires code to > explicitly call sched_bind() which should be rare. Yup. That's why I'm confused. I'm rebuilding -HEAD now with the latest flowtable changes. I'll add in my debugging afterward and trigger the particular scenario where it's behaving badly and do some more diagnostics. Thanks, -a From ache at freebsd.org Fri Feb 14 22:08:47 2014 From: ache at freebsd.org (Andrey Chernov) Date: Sat, 15 Feb 2014 02:08:38 +0400 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <201402141410.29325.jhb@freebsd.org> References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> Message-ID: <52FE93E6.6030705@freebsd.org> On 14.02.2014 23:10, John Baldwin wrote: >> Due to this bug, not fixed yet, the real picture is more complex: >> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 > > Eh, that bug report has no useful details, as in, it doesn't list the > actual commands run. If you do 'cpuset -l 6 -s 1' to force all > processes to only use CPU6, then yes, of course the other CPUs are idle > because that's what you _asked_ for. AFAICT, that is all the original > reporter did. At work we regularly add and remove CPUs from the > default set (set 1) on hundreds of machines every day with ULE without > any issues. Probably original report lack certain commands, but I provide the link to the port which reproduces this bug too. All threads there are assigned to the _different_ CPUs and appears as result on single one with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter mean too. It surely happens, maybe not the first time, but on 2nd-3rd. It means that cpuset_setaffinity() is completely broken form SCHED_ULE at least for 3 years. -- http://ache.vniz.net/ From ache at freebsd.org Fri Feb 14 22:36:24 2014 From: ache at freebsd.org (Andrey Chernov) Date: Sat, 15 Feb 2014 02:36:14 +0400 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <52FE93E6.6030705@freebsd.org> References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> Message-ID: <52FE9A5E.5050300@freebsd.org> On 15.02.2014 2:08, Andrey Chernov wrote: > On 14.02.2014 23:10, John Baldwin wrote: > >>> Due to this bug, not fixed yet, the real picture is more complex: >>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 >> >> Eh, that bug report has no useful details, as in, it doesn't list the >> actual commands run. If you do 'cpuset -l 6 -s 1' to force all >> processes to only use CPU6, then yes, of course the other CPUs are idle >> because that's what you _asked_ for. AFAICT, that is all the original >> reporter did. At work we regularly add and remove CPUs from the >> default set (set 1) on hundreds of machines every day with ULE without >> any issues. > > Probably original report lack certain commands, but I provide the link > to the port which reproduces this bug too. All threads there are > assigned to the _different_ CPUs and appears as result on single one > with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter > mean too. It surely happens, maybe not the first time, but on 2nd-3rd. > It means that cpuset_setaffinity() is completely broken form SCHED_ULE > at least for 3 years. > This is code example from cpuminer port, in case you are interested, it is very simple: static inline void affine_to_cpu(int id, int cpu) { cpuset_t set; CPU_ZERO(&set); CPU_SET(cpu, &set); cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); } ... num_processors = sysconf(_SC_NPROCESSORS_CONF); if (!opt_n_threads) opt_n_threads = num_processors; ... In the thread itself: struct thr_info *mythr = userdata; int thr_id = mythr->id; if (num_processors > 1 && opt_n_threads % num_processors == 0) { if (!opt_quiet) applog(LOG_INFO, "Binding thread %d to cpu %d", thr_id, thr_id % num_processors); affine_to_cpu(thr_id, thr_id % num_processors); } -- http://ache.vniz.net/ From adrian at freebsd.org Fri Feb 14 22:57:21 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Fri, 14 Feb 2014 14:57:19 -0800 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: References: <201402141139.49158.jhb@freebsd.org> <201402141318.44743.jhb@freebsd.org> Message-ID: [snip] So it turns out that the threads somehow migrating between CPUs during flowtable_lookup_common() is the clock swi(s), which I'm guessing are driving the per-CPU TCP callwheel timeouts. It turns out the per-CPU clock swis aren't CPU-pinned, so they can be preempted and migrated. I'm not sure if this is correct behaviour. I'll experiment with pinning these to their base CPU and see if this causes issues. Thanks, -a From avg at FreeBSD.org Sat Feb 15 00:00:07 2014 From: avg at FreeBSD.org (Andriy Gapon) Date: Sat, 15 Feb 2014 01:59:05 +0200 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <52FE9A5E.5050300@freebsd.org> References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> Message-ID: <52FEADC9.2040608@FreeBSD.org> on 15/02/2014 00:36 Andrey Chernov said the following: > On 15.02.2014 2:08, Andrey Chernov wrote: >> On 14.02.2014 23:10, John Baldwin wrote: >> >>>> Due to this bug, not fixed yet, the real picture is more complex: >>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 >>> >>> Eh, that bug report has no useful details, as in, it doesn't list the >>> actual commands run. If you do 'cpuset -l 6 -s 1' to force all >>> processes to only use CPU6, then yes, of course the other CPUs are idle >>> because that's what you _asked_ for. AFAICT, that is all the original >>> reporter did. At work we regularly add and remove CPUs from the >>> default set (set 1) on hundreds of machines every day with ULE without >>> any issues. >> >> Probably original report lack certain commands, but I provide the link >> to the port which reproduces this bug too. All threads there are >> assigned to the _different_ CPUs and appears as result on single one >> with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter >> mean too. It surely happens, maybe not the first time, but on 2nd-3rd. >> It means that cpuset_setaffinity() is completely broken form SCHED_ULE >> at least for 3 years. >> > > This is code example from cpuminer port, in case you are interested, it is very simple: > > static inline void affine_to_cpu(int id, int cpu) > { > cpuset_t set; > CPU_ZERO(&set); > CPU_SET(cpu, &set); > cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); I think that CPU_WHICH_TID should have been used here. > } > ... > num_processors = sysconf(_SC_NPROCESSORS_CONF); > if (!opt_n_threads) > opt_n_threads = num_processors; > ... > In the thread itself: > struct thr_info *mythr = userdata; > int thr_id = mythr->id; > > if (num_processors > 1 && opt_n_threads % num_processors == 0) { > if (!opt_quiet) > applog(LOG_INFO, "Binding thread %d to cpu %d", > thr_id, thr_id % num_processors); > affine_to_cpu(thr_id, thr_id % num_processors); > } > -- Andriy Gapon From jmg at funkthat.com Sat Feb 15 00:11:02 2014 From: jmg at funkthat.com (John-Mark Gurney) Date: Fri, 14 Feb 2014 16:11:00 -0800 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <52FEADC9.2040608@FreeBSD.org> References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> <52FEADC9.2040608@FreeBSD.org> Message-ID: <20140215001100.GS34851@funkthat.com> Andriy Gapon wrote this message on Sat, Feb 15, 2014 at 01:59 +0200: > on 15/02/2014 00:36 Andrey Chernov said the following: > > On 15.02.2014 2:08, Andrey Chernov wrote: > >> On 14.02.2014 23:10, John Baldwin wrote: > >> > >>>> Due to this bug, not fixed yet, the real picture is more complex: > >>>> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 > >>> > >>> Eh, that bug report has no useful details, as in, it doesn't list the > >>> actual commands run. If you do 'cpuset -l 6 -s 1' to force all > >>> processes to only use CPU6, then yes, of course the other CPUs are idle > >>> because that's what you _asked_ for. AFAICT, that is all the original > >>> reporter did. At work we regularly add and remove CPUs from the > >>> default set (set 1) on hundreds of machines every day with ULE without > >>> any issues. > >> > >> Probably original report lack certain commands, but I provide the link > >> to the port which reproduces this bug too. All threads there are > >> assigned to the _different_ CPUs and appears as result on single one > >> with SCHED_ULE (not with SCHED_4BSD). And it is what original reporter > >> mean too. It surely happens, maybe not the first time, but on 2nd-3rd. > >> It means that cpuset_setaffinity() is completely broken form SCHED_ULE > >> at least for 3 years. > >> > > > > This is code example from cpuminer port, in case you are interested, it is very simple: > > > > static inline void affine_to_cpu(int id, int cpu) > > { > > cpuset_t set; > > CPU_ZERO(&set); > > CPU_SET(cpu, &set); > > cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); > > I think that CPU_WHICH_TID should have been used here. I agree... cpuset(2): The which argument determines how the value of id is interpreted and is of type cpuwhich_t. The which argument may have the following values: CPU_WHICH_TID id is lwpid_t (thread id) CPU_WHICH_PID id is pid_t (process id) CPU_WHICH_CPUSET id is a cpusetid_t (cpuset id) CPU_WHICH_IRQ id is an irq number An id of '-1' may be used with a which of CPU_WHICH_TID, CPU_WHICH_PID, or CPU_WHICH_CPUSET to mean the current thread, process, or current thread's cpuset. All cpuset syscalls allow this usage. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From jmg at funkthat.com Sat Feb 15 00:33:52 2014 From: jmg at funkthat.com (John-Mark Gurney) Date: Fri, 14 Feb 2014 16:33:49 -0800 Subject: malloc(9) and its alignment In-Reply-To: References: <1392214788.1145.52.camel@revolution.hippie.lan> <20140212220705.GV34851@funkthat.com> Message-ID: <20140215003349.GT34851@funkthat.com> Gumpula, Suresh wrote this message on Fri, Feb 14, 2014 at 02:07 +0000: > Thanks for the explanation John. How about porting ARM way of handling required alignment with creation of separate zones > And allocating with uma_zalloc(zone) for AMD64 too for bus_dmamem_alloc? It looks like the code in HEAD is different than the code you originally quoted, this is because HEAD added support for DMAR from Intel VT-d, the code is now in bounce_bus_dmamem_alloc in x86/x86/busdma_bounce.c, but it still has the same comment: 398 /* 399 * XXX: 400 * (dmat->alignment < dmat->maxsize) is just a quick hack; the exact 401 * alignment guarantees of malloc need to be nailed down, and the 402 * code below should be rewritten to take that into account. 403 * 404 * In the meantime, we'll warn the user if malloc gets it wrong. 405 */ Per porting arm's way of handling alignment, that makes sense... Though arm's way forces all allocations to be aligned to the size of a buffer, but I don't see a better way to handle it.. > -----Original Message----- > From: John-Mark Gurney [mailto:jmg at funkthat.com] > Sent: Wednesday, February 12, 2014 5:07 PM > To: Gumpula, Suresh > Cc: Ian Lepore; freebsd-hackers at freebsd.org > Subject: Re: malloc(9) and its alignment > > Gumpula, Suresh wrote this message on Wed, Feb 12, 2014 at 19:40 +0000: > > Thanks Ian for the reply. I will look at the ARM code, but I was thinking why malloc(9) does not return bucket size aligned pointers. > > Always returning bucket sizes aligned pointers may not be ideal for a cache.. say you have a buffer of 512 bytes, where often only the first > 128 bytes are used (but all 512 bytes may be)... If you always align at 512, some cache lines will be more heavily used than others, reducing the effective size of the cache... > > This is only one reason not aligning to size may be better... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From ache at freebsd.org Sat Feb 15 01:59:34 2014 From: ache at freebsd.org (Andrey Chernov) Date: Sat, 15 Feb 2014 05:59:24 +0400 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <52FEADC9.2040608@FreeBSD.org> References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> <52FEADC9.2040608@FreeBSD.org> Message-ID: <52FEC9FC.20707@freebsd.org> On 15.02.2014 3:59, Andriy Gapon wrote: >> This is code example from cpuminer port, in case you are interested, it is very simple: >> >> static inline void affine_to_cpu(int id, int cpu) >> { >> cpuset_t set; >> CPU_ZERO(&set); >> CPU_SET(cpu, &set); >> cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); > > I think that CPU_WHICH_TID should have been used here. > You are right, thanx! -- http://ache.vniz.net/ From ache at freebsd.org Sat Feb 15 02:30:42 2014 From: ache at freebsd.org (Andrey Chernov) Date: Sat, 15 Feb 2014 06:30:38 +0400 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <20140215001100.GS34851@funkthat.com> References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> <201402141410.29325.jhb@freebsd.org> <52FE93E6.6030705@freebsd.org> <52FE9A5E.5050300@freebsd.org> <52FEADC9.2040608@FreeBSD.org> <20140215001100.GS34851@funkthat.com> Message-ID: <52FED14E.50304@freebsd.org> On 15.02.2014 4:11, John-Mark Gurney wrote: >>> This is code example from cpuminer port, in case you are interested, it is very simple: >>> >>> static inline void affine_to_cpu(int id, int cpu) >>> { >>> cpuset_t set; >>> CPU_ZERO(&set); >>> CPU_SET(cpu, &set); >>> cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); >> >> I think that CPU_WHICH_TID should have been used here. > > I agree... cpuset(2): > The which argument determines how the value of id is interpreted and is > of type cpuwhich_t. The which argument may have the following values: > > CPU_WHICH_TID id is lwpid_t (thread id) > CPU_WHICH_PID id is pid_t (process id) > CPU_WHICH_CPUSET id is a cpusetid_t (cpuset id) > CPU_WHICH_IRQ id is an irq number > > An id of '-1' may be used with a which of CPU_WHICH_TID, CPU_WHICH_PID, > or CPU_WHICH_CPUSET to mean the current thread, process, or current > thread's cpuset. All cpuset syscalls allow this usage. The question still remains: why SCHED_ULE and SCHED_4BSD do different things here on CPU_WHICH_CPUSET == -1 (current thread's cpuset)? It looks like SCHED_ULE changes per/process mask while SCHED_4BSD change per/thread mask in that case. -- http://ache.vniz.net/ From rhurlin at gwdg.de Sat Feb 15 11:29:32 2014 From: rhurlin at gwdg.de (Rainer Hurling) Date: Sat, 15 Feb 2014 12:29:21 +0100 Subject: FreeBSD UDF support In-Reply-To: References: <52FA94BF.80304@gwdg.de> <52FC8E61.5050200@FreeBSD.org> Message-ID: <52FF4F91.8060602@gwdg.de> Am 13.02.2014 16:30, schrieb Warren Block: > On Thu, 13 Feb 2014, Christian Brueffer wrote: > >> On 2/13/14 10:14 AM, Wojciech Puchar wrote: >>>> His project comes from NetBSD and modernises the UDF standard towards >>>> 2.5x. With this driver, you would be able to read modern not encrypted >>>> Blu-rays and DVDs. The driver works in principle, some minor issues >>>> have >>>> to be solved. >>>> >>>> Unfortunately there is only very little interest to integrate this >>>> driver into base. >>>> >>> i need RW support so it is not much use for me. >> >> NetBSD has newfs_udf(8), but it doesn't look like anyone has ported it >> to FreeBSD yet, > > GSOC candidates have been requested. That might be perfect. I don't know when it was added, but it is always listed as No #6: https://wiki.freebsd.org/SummerOfCode2014#Port_NetBSD.27s_UDF_implementation From perryh at pluto.rain.com Sun Feb 16 06:13:12 2014 From: perryh at pluto.rain.com (Perry Hutchison) Date: Sat, 15 Feb 2014 21:16:17 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: Message-ID: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> Jordan Hubbard wrote: > Even variant symlinks (/bin -> /${ARCH}/bin), which can expand > differently depending on the user context, have clearly > understandable semantics - you know that the symlink is going > to expand to exactly one file no matter what ARCH is set to. s/file/pathname/ Depending on what ARCH is set to, the expanision may or may not point to any actual file (or directory, or ...) From ilya at bakulin.de Sun Feb 16 11:11:55 2014 From: ilya at bakulin.de (Ilya Bakulin) Date: Sun, 16 Feb 2014 12:11:53 +0100 Subject: MMC/SDIO stack under CAM Message-ID: <20140216111153.GA74858@olymp.kibab.com> Hi list, so I still want to move SDIO stack integration forward. As was already discussed, the best thing to do would be to have MMC stack under CAM. I have tried to understand how the CAM works for the existing drivers. Below is the structure for USB sticks: +-----------------------+ |Kernel (disk interface)| +-----------------------+ | BIO | +-------------------+ |da (storage driver)| +-------------------+ | CCB | +------------------------+ |CAM Layer sys/cam/scsi/*| +------------------------+ | CCB | +------------------+ |umass (HBA == SIM)| +------------------+ | usbd_* | +--------------------------+ |USB stack (and controller)| +--------------------------+ So da(4) doesn't know anything about USB. At this point I thought, that in this case da(4) will happily communicate with MMC/SD cards, if we provide a SIM for MMC! So the structure for MMC would be like this: +-----------------------+ |Kernel (disk interface)| +-----------------------+ | BIO | +-------------------+ |da (storage driver)| +-------------------+ | CCB | +------------------------+ |CAM Layer (sys/cam/mmc/*| +------------------------+ | CCB | +-------------------+ |mmc_cam (HBA == SIM| +-------------------+ | SD/MMC protocol (struct mmc_request) | +-------------------------------------------+ | MMC ctrlr driver (sdhci_ti, ..., mmcnull) | +-------------------------------------------+ | | +------------------+ | MMC/SD/SDIO Card | +------------------+ (the mmcnull driver is the pseudo-driver that I'm writing to make it possible to develop and test the whole thing on the VM). So MMC SIM and MMC controller driver would exchange mmc_requests, as it was before, with the exception that command completion will be reported differently (to utilize async callbacks system of CAM). For SDIO card, the situation will be different. Essentially, we should allow a device driver to send arbitrary messages to the card. So the device driver will attach directly to the scbus. Please let me know your thoughts about it. I really want SDIO make its way into the kernel, because there is an increasing number of ARM boards on the market that have integrated SDIO WLAN on them and we cannot support them fully. -- Ilya -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From imp at bsdimp.com Sun Feb 16 18:05:45 2014 From: imp at bsdimp.com (Warner Losh) Date: Sun, 16 Feb 2014 11:05:35 -0700 Subject: MMC/SDIO stack under CAM In-Reply-To: <20140216111153.GA74858@olymp.kibab.com> References: <20140216111153.GA74858@olymp.kibab.com> Message-ID: <5C2CF572-360D-4CA0-81C7-18A5C455AED5@bsdimp.com> On Feb 16, 2014, at 4:11 AM, Ilya Bakulin wrote: > Hi list, > so I still want to move SDIO stack integration forward. > As was already discussed, the best thing to do would be to > have MMC stack under CAM. > I have tried to understand how the CAM works for the existing drivers. > > Below is the structure for USB sticks: > > +-----------------------+ > |Kernel (disk interface)| > +-----------------------+ > | > BIO > | > +-------------------+ > |da (storage driver)| > +-------------------+ > | > CCB > | > +------------------------+ > |CAM Layer sys/cam/scsi/*| > +------------------------+ > | > CCB > | > +------------------+ > |umass (HBA == SIM)| > +------------------+ > | > usbd_* > | > +--------------------------+ > |USB stack (and controller)| > +--------------------------+ > > So da(4) doesn't know anything about USB. > At this point I thought, that in this case da(4) > will happily communicate with MMC/SD cards, if we provide > a SIM for MMC! So the structure for MMC would be like this: > > +-----------------------+ > |Kernel (disk interface)| > +-----------------------+ > | > BIO > | > +-------------------+ > |da (storage driver)| > +-------------------+ > | > CCB > | > +------------------------+ > |CAM Layer (sys/cam/mmc/*| > +------------------------+ > | > CCB > | > +-------------------+ > |mmc_cam (HBA == SIM| > +-------------------+ > | > SD/MMC protocol (struct mmc_request) > | > +-------------------------------------------+ > | MMC ctrlr driver (sdhci_ti, ..., mmcnull) | > +-------------------------------------------+ > | > | > +------------------+ > | MMC/SD/SDIO Card | > +------------------+ > > (the mmcnull driver is the pseudo-driver that I'm writing > to make it possible to develop and test the whole thing > on the VM). that's cool! I'd thought of writing a mmcsim that I could use to develop the stack on x86, but time pressures of my job precluded that (though in hind sight, it would have saved a lot of time). > So MMC SIM and MMC controller driver would exchange mmc_requests, > as it was before, with the exception that command completion will be > reported differently (to utilize async callbacks system of CAM). > > For SDIO card, the situation will be different. Essentially, > we should allow a device driver to send arbitrary messages to the card. > So the device driver will attach directly to the scbus. > > Please let me know your thoughts about it. > I really want SDIO make its way into the kernel, because there > is an increasing number of ARM boards on the market that have integrated > SDIO WLAN on them and we cannot support them fully. Generally, I like this idea... I'd be very interested in some of the specifics to make the direct attachment work with SDIO. That's the one area I either don't think I understand your proposal well enough, or I do and I'm concerned about it. :) So maybe write a bit more about the details of the SDIO cards and how they's interact with CAM in this scenario... The notion of moving MMC/SD into the CAM stack has been around since I started working on MMC. At the time, CAM was too SCSI centric to do it, but Alexander Motin's work has generally improved that situation, so it may be possible now to do it and shake out the few dark corners of CAM where this isn't the case. I also think it should help performance a lot since we're currently single threaded to the card (but that's also the nature of the bus), which introduces some round-trip latencies between too many threads that CAM may help us avoid. Warner From trociny at FreeBSD.org Sun Feb 16 21:58:48 2014 From: trociny at FreeBSD.org (Mikolaj Golub) Date: Sun, 16 Feb 2014 23:58:43 +0200 Subject: The sonewconn listen queue overflow issue In-Reply-To: <52EFEEF7.5010704@xs4all.nl> References: <52EFEEF7.5010704@xs4all.nl> Message-ID: <20140216215842.GB14237@gmail.com> On Mon, Feb 03, 2014 at 08:33:11PM +0100, Photo stuff wrote: > sonewconn: pcb 0xyyyyyyyyyyyyyyyy: Listen queue overflow: 8 already in > queue awaiting acceptance > > I searched a bit on the web and came across recommendations to try > netstat -nAa to find out which program this came from. > > Well, running netstat -nAa |grep pcb 0xyyyyyyyyyyyyyyyy in a loop didn't > work, it didn't give any output even though the messages kept coming in > the log during that time. Unfortunately, netstat(1) shows tcpcb address for TCP sockets, while in debug messages like above the socket's pcb is printed. It looks like the simplest way to assiciate a socket with the reported pcb address is to hack netstat(1) to output pcb instead of tcpcb: Index: usr.bin/netstat/inet.c =================================================================== --- usr.bin/netstat/inet.c (revision 261639) +++ usr.bin/netstat/inet.c (working copy) @@ -448,7 +448,7 @@ protopr(u_long off, const char *name, int af1, int if (Lflag && so->so_qlimit == 0) continue; if (Aflag) { - if (istcp) + if (0 && istcp) printf("%*lx ", 2 * (int)sizeof(void *), (u_long)inp->inp_ppcb); else printf("%*lx ", 2 * (int)sizeof(void *), (u_long)so->so_pcb); -- Mikolaj Golub From torek at torek.net Mon Feb 17 01:12:42 2014 From: torek at torek.net (Chris Torek) Date: Sun, 16 Feb 2014 18:12:35 -0700 Subject: The sonewconn listen queue overflow issue In-Reply-To: Your message of "Sun, 16 Feb 2014 23:58:43 +0200." <20140216215842.GB14237@gmail.com> Message-ID: <201402170112.s1H1CZlR064210@elf.torek.net> >Unfortunately, netstat(1) shows tcpcb address for TCP sockets, >while in debug messages like above the socket's pcb is printed. This happens because, in the place where the listen queue overflow occurs, we do not have access to that which would allow identifying the specific overflowing queue. Specifically, we only know the socket, and a PCB for it (so_pcb), and that happens to be an inpcb rather than a tcpcb, in the case of a TCP socket. (I'm not sure why the socket pcb is the inpcb in the first place. Seems like the whole system could have been designed to work from top down to bottom: so_pcb is the tcp or udp block, and from there we get the inpcb, etc. [This would match so_proto being the TCP or UDP protocol, etc.] But it wasn't, and reversing that is too painful now.) >It looks like the simplest way to assiciate a socket with the >reported pcb address is to hack netstat(1) to output pcb instead >of tcpcb: [snip patch] That's kind of going backwards though: dumbing down netstat just to handle some deliberate ignorance in the kernel. :-) I think a better fix might be to have a "report listen-queue overflow" function in the protocol (so_proto or its pr_usrreqs -- season to taste). For tcp sockets this could log the local address; other protocols would log something else as appropriate; and For backwards compatibility, if there is no reporting function, the existing code can remain as-is. Chris From jhellenthal at dataix.net Mon Feb 17 04:47:12 2014 From: jhellenthal at dataix.net (Jason Hellenthal) Date: Sun, 16 Feb 2014 23:47:07 -0500 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> Message-ID: <1D850D2F-4E1F-4958-8EC9-1774A1793C60@dataix.net> That is a rather lovely example. Going a bit beyond my initial thoughts. . . . this could be really handy if done correctly. For ports, source, object tree's . . . obviously bin directories and even lib directories. Call it the poor-mans replication system but with some pretty big benifits. My initial thoughts were to just point directly to files, excluding using globing as a way to just include files in a massive config sort of fashion but if globing comes into play that would be awesome. -- Jason Hellenthal Voice: 95.30.17.6/616 JJH48-ARIN > On Feb 16, 2014, at 0:16, perryh at pluto.rain.com (Perry Hutchison) wrote: > > Jordan Hubbard wrote: > >> Even variant symlinks (/bin -> /${ARCH}/bin), which can expand >> differently depending on the user context, have clearly >> understandable semantics - you know that the symlink is going >> to expand to exactly one file no matter what ARCH is set to. > > s/file/pathname/ > > Depending on what ARCH is set to, the expanision may or may not > point to any actual file (or directory, or ...) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6118 bytes Desc: not available URL: From noname.esst at yahoo.com Mon Feb 17 09:55:35 2014 From: noname.esst at yahoo.com (Nomad Esst) Date: Mon, 17 Feb 2014 01:52:38 -0800 (PST) Subject: ifconfig siocifcreate invalid argument Message-ID: <1392630758.77600.YahooMailNeo@web162704.mail.bf1.yahoo.com> Hi Recently I've customized my kernel and deleted most of unused devices and options. When I try to create a a gre interface I get this error: ifconfig: siocifcreate2: invalid argument I'm using FreeBSD9.2 AMD 64 What's the problem? From freebsd-hackers at dino.sk Mon Feb 17 10:15:49 2014 From: freebsd-hackers at dino.sk (Milan Obuch) Date: Mon, 17 Feb 2014 11:10:35 +0100 Subject: ifconfig siocifcreate invalid argument In-Reply-To: <1392630758.77600.YahooMailNeo@web162704.mail.bf1.yahoo.com> References: <1392630758.77600.YahooMailNeo@web162704.mail.bf1.yahoo.com> Message-ID: <20140217111035.328baa3f@zeta.dino.sk> On Mon, 17 Feb 2014 01:52:38 -0800 (PST) Nomad Esst wrote: > Hi > Recently I've customized my kernel and deleted most of unused devices > and options. When I try to create a a gre interface I get this error: > > > ifconfig: siocifcreate2: invalid argument > > I'm using FreeBSD9.2 AMD 64 > > What's the problem? > If you do not have 'device gre' in kernel config and module if_gre is not loaded automatically, you need to do it yourself - just use 'kldload if_gre' before you try to create interface greN. Regards, Milan From noname.esst at yahoo.com Mon Feb 17 10:25:53 2014 From: noname.esst at yahoo.com (Nomad Esst) Date: Mon, 17 Feb 2014 02:22:53 -0800 (PST) Subject: ifconfig siocifcreate invalid argument In-Reply-To: <20140217111035.328baa3f@zeta.dino.sk> References: <1392630758.77600.YahooMailNeo@web162704.mail.bf1.yahoo.com> <20140217111035.328baa3f@zeta.dino.sk> Message-ID: <1392632573.91721.YahooMailNeo@web162705.mail.bf1.yahoo.com> Thanks Do you know which kernel module should be included in kernel config file? Since we could create gre interfaces without any problems before we customize the kernel. On Monday, February 17, 2014 1:40 PM, Milan Obuch wrote: On Mon, 17 Feb 2014 01:52:38 -0800 (PST) > >Nomad Esst wrote: > >> Hi >> Recently I've customized my kernel and deleted most of unused devices >> and options. When I try to create a a gre interface I get this error: >> >> >> ifconfig: siocifcreate2: invalid argument >> >> I'm using FreeBSD9.2 AMD 64 >> >> What's the problem? >> > >If you do not have 'device gre' in kernel config and module if_gre is >not loaded automatically, you need to do it yourself - just use >'kldload if_gre' before you try to create interface greN. > >Regards, >Milan > > > > From freebsd-hackers at dino.sk Mon Feb 17 13:06:28 2014 From: freebsd-hackers at dino.sk (Milan Obuch) Date: Mon, 17 Feb 2014 14:06:25 +0100 Subject: ifconfig siocifcreate invalid argument In-Reply-To: <1392632573.91721.YahooMailNeo@web162705.mail.bf1.yahoo.com> References: <1392630758.77600.YahooMailNeo@web162704.mail.bf1.yahoo.com> <20140217111035.328baa3f@zeta.dino.sk> <1392632573.91721.YahooMailNeo@web162705.mail.bf1.yahoo.com> Message-ID: <20140217140625.7f18d14f@zeta.dino.sk> On Mon, 17 Feb 2014 02:22:53 -0800 (PST) Nomad Esst wrote: > Thanks > Do you know which kernel module should be included in kernel config > file? Since we could create gre interfaces without any problems > before we customize the kernel. > Well, no idea - I just try and did not see this error on any system here, checked on 9.2-STABLE/amd64 as well, all with minimal kernel config (no network card in kernel, no SCSI adapter in kernel, no sound card in kernel etc.) and every time if_gre kernel module was loaded correctly when 'ifconfig gre0 create' command was issued in shell. Maybe your ifconfig needs to be rebuilt if you have not build system and binaries from the same sources, but no other idea here... Also, in my sources I did not find 'device gre' in kernel either, so I am not sure if this possibility even exists, but you can try it just to be sure... > On Monday, February 17, 2014 1:40 PM, Milan Obuch > wrote: > On Mon, 17 Feb 2014 01:52:38 -0800 (PST) > > > >Nomad Esst wrote: > > > >> Hi > >> Recently I've customized my kernel and deleted most of unused > >> devices and options. When I try to create a a gre interface I get > >> this error: > >> > >> > >> ifconfig: siocifcreate2: invalid argument > >> > >> I'm using FreeBSD9.2 AMD 64 > >> > >> What's the problem? > >> > > > >If you do not have 'device gre' in kernel config and module if_gre is > >not loaded automatically, you need to do it yourself - just use > >'kldload if_gre' before you try to create interface greN. > > > >Regards, > >Milan > > Milan From erichsfreebsdlist at alogt.com Mon Feb 17 13:31:41 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Mon, 17 Feb 2014 21:31:27 +0800 Subject: ifconfig siocifcreate invalid argument In-Reply-To: <20140217140625.7f18d14f@zeta.dino.sk> References: <1392630758.77600.YahooMailNeo@web162704.mail.bf1.yahoo.com> <20140217111035.328baa3f@zeta.dino.sk> <1392632573.91721.YahooMailNeo@web162705.mail.bf1.yahoo.com> <20140217140625.7f18d14f@zeta.dino.sk> Message-ID: <20140217213127.41281a83@X220.alogt.com> Hi, On Mon, 17 Feb 2014 14:06:25 +0100 Milan Obuch wrote: > On Mon, 17 Feb 2014 02:22:53 -0800 (PST) > Nomad Esst wrote: > > > Thanks > > Do you know which kernel module should be included in kernel config > > file? Since we could create gre interfaces without any problems > > before we customize the kernel. > > > > Well, no idea - I just try and did not see this error on any system > here, checked on 9.2-STABLE/amd64 as well, all with minimal kernel > config (no network card in kernel, no SCSI adapter in kernel, no sound > card in kernel etc.) and every time if_gre kernel module was loaded > correctly when 'ifconfig gre0 create' command was issued in shell. my observation is that if the kernel contains a certain module A which required module B to run, module B is not loaded. Stripping all modules which might be required out of the kernel and then get them loaded later will show with kldstat whet modules have been loaded. Of course, modules not needed yet are not shown. If there is no comment in GENERIC, it is a bit of trial and error to find the modules needed. Erich From wkoszek at freebsd.org Tue Feb 18 07:29:23 2014 From: wkoszek at freebsd.org (Wojciech A. Koszek) Date: Tue, 18 Feb 2014 07:28:21 +0000 Subject: BSD XXI Manifesto Message-ID: <20140218072821.GF34282@FreeBSD.org> (cross-posted message: eventual discussion let's keep on hackers@) Hello, After being disappointed with the list of submitted FreeBSD ideas, I created my own Machiavellist vision of XXI-century FreeBSD. I paste it below. If you want to add something, it's here: https://wiki.freebsd.org/BSD_XXI_Manifesto GSOC students could use this as an inspiration for their projects. The idea is to invite non-C, non-OS, non-kernel developers to help out with FreeBSD stuff. ============ BSDXXI manifesto For people seeking for inspiration here are some ideas of how FreeBSD of 21st century could operate... - FreeBSD boots from simple USB-stick image or SD card. Image would contain nothing more than HTTP-enabled installer, so that as OS updates are rolled out, the installer doesn't change. It means I can keep the same USB-stick for years, since all packages/installation procedure would actually be fetched from the Internet. - installer is in high-resolution mode with graphical environment started, and my bluetooth/remote-USB mouse/keyboard working OK. Additionally I have an access to the filesystem on USB-stick. I have Wi-Fi access and/or Ethernet access and a browser to do the installation. This all is in case I want to bother to use a keyboard. - installer starts HTTP server in the background. It serves responsive 'FreeBSD installer' website. You can enter this website with your smartphone and perform the whole installation procedure from your mobile device. This HTTP server is getting left of "service partition", so that in case of problems, you can always boot FreeBSD in emergency mode and recover from problems. - installer asks me whether I'd like to use my FreeBSD account or to setup one. FreeBSD is similar to Apple ID or Amazon ID and remembers my preferences, so that in case of installation/backup/recovery I don't have to be asked same questions 10 times. This data is stored in an encrypted form on my disk, and gets imported by all programs that may ever want to ask me about my name/surname etc. This account is used to submit data about hardware which I use too, so that developers see which hardware is popular among users. This account is used for encrypted crashdump reports, bug reporting and others. This is the key to the contact with the FreeBSD developers community. Also as a part of the account users get easy access to 'BSDDEV' environment, which lets them clone, change and contribute back changes by themselves, without asking any commiters/developers for anything. They may later request this change to be pushed back to the FreeBSD. - installer asks me whether I'd like to pick certain profile of installation and install everything in 1 step. This is similar to "1 click buy" on Amazon, and is similar to node.js "npm" manager -- dozens of users contribute their installation profiles to the FreeBSD portal, and FreeBSD accounts have access to these profiles. Everybody sees which installation profile has most "Likes", and based on that users can choose whether to struggle with 100 parameters for the installation, or pick something that expert picked for them. Also desktop users willing to use Flash can pick "freebsd_superflash" installation profile, which will give them Flash working out of the box etc... - Among installation profiles, installer presents me: * mobile installation (cell phone) FreeBSD of XXI century runs on most of mobile devices, and this is one of the types of installation which I can pick for my tablet/smartphone. FreeBSD found great adoption in mobile environments, since BSD license attracted more people than GPLv4 code. FreeBSD has drivers for all sensors, GPSes and GSM/UMTS modules from phones. * laptop This profile installs me all possible optimizations for maximal power and maximal battery life, and minimal noise for any leftover laptops that still have a fan. * desktop This profile installs me everything for a desktop user. * VM (cloud solution: EC2, Rackspace etc) This installation may pick optimized I/O schedulers, so that environments which charge for I/O access, network access etc.. end up being cheap thanks to smart algorithms in FreeBSD. * hosted cloud This is a profile for enterprise users. Hosted installation is something which lets you designate a master, and slaves which connect to the master. Once slaves are connected, from 1 installer you can choose what all nodes will get installed and how they'll work. So e.g.: if you want to have a storage-cloud with 4 servers, you can make them be a redundant storage with 2 servers each, for improved efficiency. If servers are plugged in the same switch, I'd like to see them all in the menu and be able to toggle the ones which I want to be added to the cluster. - FreeBSD is getting setup in less than 5 minutes. All stages are marked with barcodes/URLs, so that in case of problems, user can take a photo of a screen and get immediate helps from FreeBSD community portal. - FreeBSD on 1st boot establishes connection via your FreeBSD ID and submits stuff necessary for improved FreeBSD development. You may choose not to submit stuff, but you'll not get an access to interactive bug database, easy contribution capability etc.. - FreeBSD ID offers to sync my disk with the BSDCloud, and offers me options for storage providers and their prices. - FreeBSD once installed, just works. It includes only the most necessary ports installed as a part of chosen profile. If necessary, users can share their configurations, e.g.: if there's somebody who got Jail/MAC/Capability-enabled environment for Node.Js installed in /jail/nodejs/, I don't really want to keep retyping his commands like a monkey. I just want to get his configuration and 20s is max I can wait for this to happen. This includes things which are popular, but boring, e.g.: HTTP server, SQL server, Mail server etc.. configuration. I'd like to add my 3 domains: koszek.com freebsd.czest.pl something.com to FreeBSD system, pick my HTTP solution, my Mail solution, my SQL solution and have them installed in the most security-hardened way possible with 0 effort. - I keep all of important system actions versioned/logged, so that if I happen to have some problems with ports/packages, I can send the journal of what happened to my system, so that somebody can reproduce it. - in cases of problems with programs, I screen share my terminal with other FreeBSD IDs. I'd like to say: "I want FreeBSD ID 'cognet' to have access to this computer but only for watching", so that I can show the problem to the 2nd FreeBSD developer. It should also be possible to give full-access to the machine to the developer, so that in case of critical problems developer can reproduce exactly what's going on. - FreeBSD rarely asks me to compile anything. FreeBSD.org has powerful BSDCloud configuration which compiles everything for me. So if I want custom kernel configuration, I can mark checkboxes online and BSDCloud will compile the kernel quickly for me, so that I don't have to bother with it. - In case of problems, it's very easily to submit a record. FreeBSD loader, FreeBSD kernel and FreeBSD user-space utilities are all equiped with OCR-enabled/scannable mechanism, which lets me to use my phone to submit a bug report. It should be possible to e.g.: take a photo of a screen, have your phone recognize what the photo is all about and act accordingly via your FreeBSD ID (submit benchmark result, submit bug report, submit security problem etc..) - FreeBSD developers check in stuff to the repository. Upon check-in FreeBSD gets compiled and automatically booted on all possible platforms -- where possible in the BSDCloud, and where it's not possible, it gets booted on real hardware. Developer has access to the BSDCloud, so should the problem happen, the VM is frozen and dev. can login to it and see what's wrong. - Upon each commit FreeBSD regression test suite is run. Test suite tries to make sure FreeBSD is still runnable, and whether any regressions got introduced. Each unit test lets you to start, stop and modify existing VM by typing commands in it and comparing the result the expected one. Users can easily contribute to regression tests by recording their actions and contributing the recorded sessions back: their FreeBSD IDs can submit regression sets to the repository and their tests will be included in the next regression run. - FreeBSD documentation is always up-to-date. Regression suite is basically a specialized Domain Specific Language, that is especially annotated with comments which are an integral part of the code. This code is used to generate the documentation -- upon each action the screenshot is taken and explanation gets inserted. Once regression test is run, and screenshots are obtained, they get glued together for a slide-show (HTML page) or screencast and are being exported to YouTube. - Check-in process gets modified, so that only when all accepted regression sets are run, the check-in is accepted. - Documentation is available in easy-to-modify form for all FreeBSD IDs. Internal format for the documentation doesn't matter, since everything gets edited in the WWW browser anyway. - Once the document is submitted, it gets reviewed and accepted by the FreeBSD developer. Upon commit, all FreeBSD IDs whose users marked 'want to contribute to documentation' get the notification on document change. The ones which speak other languages get a chance to translate the changes right away and earn credits. - submission of regression test/patch/doc change earns FreeBSD ID certain reputation. FreeBSD ID could get tied to FreeBSD forums, so that people who help others a lot also get credits. There's a simple 'acceptance' formula: above X credits you get privileges A, B, C within FreeBSD.org. - FreeBSD development can happen entirely online, via BSDCould. Users and developers can edit files via WWW browser and do it the same way. They compile the system and boot it in VM and later, on the real hardware. - Upon making a change, I select 'users who have device X' and users who marked the checkbox 'Willing to test'. They get the VM which I used for testing available and can clone the VM's configuration to their system with 1 click, boot on their hardware and report the results. - Donation process gets modified so that users who care about certain devices a lot can send their hardware to BSDCloud. Hardware would get plugged in the physical hardware and since then regression tests testing this piece of hardware would be run on each commit. Expensive hardware can get linked with BSDCloud so that the machine stays on the owners side. This box is available via VPN just like any other BSDCloud box, and as long as it's available, regression suite is run on it as well. VPN works across firewall and proxies, so specialized platforms behind the corporate walls can also get tested. - On each commit set of benchmarks is run and visualized in the browser. The configuration can include the VM configurations, but can also involve hardware. So before performing a change, developer can see the impact of the change on the system performance. - On each commit set of power benchmarks is run. Couple of real hardware setups have power measurement attached to them and are able to export power profiling information upon commit. This is crucial for cell phones, which FreeBSD can run. ============ -- Wojciech A. Koszek wkoszek at FreeBSD.czest.pl http://FreeBSD.czest.pl/~wkoszek/ From mail at ma17.ata.myota.org Tue Feb 18 18:09:41 2014 From: mail at ma17.ata.myota.org (Andre Albsmeier) Date: Tue, 18 Feb 2014 19:06:46 +0100 Subject: pthread programming eats up resources (My or FreeBSD's fault?) Message-ID: <20140218180646.GA67861@schlappy> Well, as these are my first steps regarding thread programming, it's probably my fault... Why does this programme slowly grow and grow until it hits resource limits? ----- snip pth1.c ----- #include #include #include void* mythread( void* arg ) { return NULL; } int main( int argc, const char* const argv[] ) { pthread_t pthr; int i; while( 1 ) { for( i=1000; i; i-- ) if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) fprintf( stderr, "pthread_create\n" ); else pthread_detach( pthr ); putchar( '.' ); fflush( stdout ); usleep( 25000 ); } } ----- snap ----- Just to be sure I have also created the non-detaching version which behaves in the same way: ----- snip pth2.c ----- #include #include #include #define M 1000 pthread_t pthr[M]; void* mythread( void* arg ) { return NULL; } int main( int argc, const char* const argv[] ) { int i; while( 1 ) { for( i=M; i; i-- ) if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 ) fprintf( stderr, "pthread_create\n" ); for( i=M; i; i-- ) if( pthread_join( pthr[i], NULL ) != 0 ) fprintf( stderr, "pthread_join\n" ); putchar( '.' ); fflush( stdout ); usleep( 25000 ); } } ----- snap ----- Compile them using -pthread and watch their ps output in another window (FreeBSD-9.2 but that shouldn't matter). So what am I doing wrong here? Thanks, -Andre From mail at ma17.ata.myota.org Tue Feb 18 19:21:37 2014 From: mail at ma17.ata.myota.org (Andre Albsmeier) Date: Tue, 18 Feb 2014 20:21:19 +0100 Subject: GEOM mentor request In-Reply-To: <20131101103158.GA35397@lemon> References: <20131101103158.GA35397@lemon> Message-ID: <20140218192119.GA86251@schlappy> On Fri, 01-Nov-2013 at 10:31:58 +0000, symbolics at gmx.com wrote: > Hi hackers, > > I have been studying the GEOM documentation and source recently. Is > anyone actively maintaining this subsystem at the moment? I would like > to give it some attention. A few example things I'd like to work on (in > no particular order): > ... > + Probably other bits I can't remember right now. Well, in case you are looking for a probably easy task: Teach gstripe and geli about TRIM as it was done with gmirror in http://svnweb.freebsd.org/base?view=revision&revision=237930 http://svnweb.freebsd.org/base?view=revision&revision=237929 I wanted to have a look a this since more than one year now but time never permitted it (and that's why I am replying to your message after more than 3 months ;-)). -Andre From jhb at freebsd.org Tue Feb 18 19:58:24 2014 From: jhb at freebsd.org (John Baldwin) Date: Tue, 18 Feb 2014 14:42:36 -0500 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <52FED14E.50304@freebsd.org> References: <20140215001100.GS34851@funkthat.com> <52FED14E.50304@freebsd.org> Message-ID: <201402181442.36380.jhb@freebsd.org> On Friday, February 14, 2014 9:30:38 pm Andrey Chernov wrote: > On 15.02.2014 4:11, John-Mark Gurney wrote: > >>> This is code example from cpuminer port, in case you are interested, it is very simple: > >>> > >>> static inline void affine_to_cpu(int id, int cpu) > >>> { > >>> cpuset_t set; > >>> CPU_ZERO(&set); > >>> CPU_SET(cpu, &set); > >>> cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_CPUSET, -1, sizeof(cpuset_t), &set); > >> > >> I think that CPU_WHICH_TID should have been used here. > > > > I agree... cpuset(2): > > The which argument determines how the value of id is interpreted and is > > of type cpuwhich_t. The which argument may have the following values: > > > > CPU_WHICH_TID id is lwpid_t (thread id) > > CPU_WHICH_PID id is pid_t (process id) > > CPU_WHICH_CPUSET id is a cpusetid_t (cpuset id) > > CPU_WHICH_IRQ id is an irq number > > > > An id of '-1' may be used with a which of CPU_WHICH_TID, CPU_WHICH_PID, > > or CPU_WHICH_CPUSET to mean the current thread, process, or current > > thread's cpuset. All cpuset syscalls allow this usage. > > The question still remains: why SCHED_ULE and SCHED_4BSD do different > things here on CPU_WHICH_CPUSET == -1 (current thread's cpuset)? It > looks like SCHED_ULE changes per/process mask while SCHED_4BSD change > per/thread mask in that case. Eh, no. CPU_WHICH_CPUSET changes the global set that the thread belongs to. t is not per proceses or per thread. Unless you are explicitly creating new global sets, your thread belongs to the default set (set 1), and this call is changing the default set (set 1) to only use a single CPU. This affects all processes in the machine that do not have their own sets. -- John Baldwin From ache at freebsd.org Wed Feb 19 14:35:26 2014 From: ache at freebsd.org (Andrey Chernov) Date: Wed, 19 Feb 2014 18:29:01 +0400 Subject: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? In-Reply-To: <201402181442.36380.jhb@freebsd.org> References: <20140215001100.GS34851@funkthat.com> <52FED14E.50304@freebsd.org> <201402181442.36380.jhb@freebsd.org> Message-ID: <5304BFAD.50502@freebsd.org> On 18.02.2014 23:42, John Baldwin wrote: > Eh, no. CPU_WHICH_CPUSET changes the global set that the thread belongs to. > t is not per proceses or per thread. Unless you are explicitly creating > new global sets, your thread belongs to the default set (set 1), and this > call is changing the default set (set 1) to only use a single CPU. This > affects all processes in the machine that do not have their own sets. > Thanx for detailed explanation. According to it, SCHED_ULE does the right thing for such CPU_WHICH_CPUSET call, while SCHED_4BSD does not. Is recent r260043 supposed to fix the problem for SCHED_4BSD or is it unrelated? I can't check it by myself yet. -- http://ache.vniz.net/ From mail at ma17.ata.myota.org Wed Feb 19 14:47:42 2014 From: mail at ma17.ata.myota.org (Andre Albsmeier) Date: Wed, 19 Feb 2014 15:47:28 +0100 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140218180646.GA67861@schlappy> References: <20140218180646.GA67861@schlappy> Message-ID: <20140219144728.GA3036@schlappy> [Commenting my own mail from below and CC'ing freebsd-threads.] I have tested my code below on a Linux box (3.2.0) and here the behaviour is different and seems correct: While watching with top(1), VIRT climbs up to a few GB and collapses then to a few MB (somehow reminding me of some kind of garbage collection). Important thing is that RES always stays below 1MB. On FreeBSD SIZE and RES are about 100MB apart but both are increasing util 2GB and pth1 dies with Cannot map anonymous memory Out of memory So the question is: Is my programme buggy and Linux works around this bug or is there some kind of memory leak in the pthreads code in FreeBSD? -Andre ----- Forwarded message ----- Well, as these are my first steps regarding thread programming, it's probably my fault... Why does this programme slowly grow and grow until it hits resource limits? ----- snip pth1.c ----- #include #include #include void* mythread( void* arg ) { return NULL; } int main( int argc, const char* const argv[] ) { pthread_t pthr; int i; while( 1 ) { for( i=1000; i; i-- ) if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) fprintf( stderr, "pthread_create\n" ); else pthread_detach( pthr ); putchar( '.' ); fflush( stdout ); usleep( 25000 ); } } ----- snap ----- Just to be sure I have also created the non-detaching version which behaves in the same way: ----- snip pth2.c ----- #include #include #include #define M 1000 pthread_t pthr[M]; void* mythread( void* arg ) { return NULL; } int main( int argc, const char* const argv[] ) { int i; while( 1 ) { for( i=M; i; i-- ) if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 ) fprintf( stderr, "pthread_create\n" ); for( i=M; i; i-- ) if( pthread_join( pthr[i], NULL ) != 0 ) fprintf( stderr, "pthread_join\n" ); putchar( '.' ); fflush( stdout ); usleep( 25000 ); } } ----- snap ----- Compile them using -pthread and watch their ps output in another window (FreeBSD-9.2 but that shouldn't matter). So what am I doing wrong here? Thanks, -Andre From erichsfreebsdlist at alogt.com Wed Feb 19 15:08:34 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Wed, 19 Feb 2014 23:08:24 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140219144728.GA3036@schlappy> References: <20140218180646.GA67861@schlappy> <20140219144728.GA3036@schlappy> Message-ID: <20140219230824.0f2ba24b@X220.alogt.com> Hi, as I understand your program, it creates 1000 threads, waits a but and then starts again creating 1000 threads until something kill it. From my point of view, your program depends very much on the default settings of the environment. If the environment allows the immediate execution of the new thread, you will never get many threads. Otherwise, the number of threads hanging around could add up. It also depends on the number of CPUs/cores your system has. But your are right, it should not crash on a modern machine but it still could use some amount of memory. Erich On Wed, 19 Feb 2014 15:47:28 +0100 Andre Albsmeier wrote: > [Commenting my own mail from below and CC'ing freebsd-threads.] > > I have tested my code below on a Linux box (3.2.0) and here the > behaviour is different and seems correct: While watching with > top(1), VIRT climbs up to a few GB and collapses then to a few > MB (somehow reminding me of some kind of garbage collection). > Important thing is that RES always stays below 1MB. > > On FreeBSD SIZE and RES are about 100MB apart but both are > increasing util 2GB and pth1 dies with > > Cannot map anonymous memory > Out of memory > > So the question is: Is my programme buggy and Linux works around > this bug or is there some kind of memory leak in the pthreads > code in FreeBSD? > > -Andre > > ----- Forwarded message ----- > > Well, as these are my first steps regarding thread programming, > it's probably my fault... > > Why does this programme slowly grow and grow until it hits > resource limits? > > ----- snip pth1.c ----- > > #include > #include > #include > > void* mythread( void* arg ) > { > return NULL; > } > > int main( int argc, const char* const argv[] ) > { > pthread_t pthr; > int i; > > while( 1 ) { > > for( i=1000; i; i-- ) > if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) > fprintf( stderr, "pthread_create\n" ); > else > pthread_detach( pthr ); > > putchar( '.' ); > fflush( stdout ); > usleep( 25000 ); > } > } > > ----- snap ----- > > Just to be sure I have also created the non-detaching version > which behaves in the same way: > > ----- snip pth2.c ----- > > #include > #include > #include > > #define M 1000 > > pthread_t pthr[M]; > > void* mythread( void* arg ) > { > return NULL; > } > > int main( int argc, const char* const argv[] ) > { > int i; > > while( 1 ) { > > for( i=M; i; i-- ) > if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 ) > fprintf( stderr, "pthread_create\n" ); > > for( i=M; i; i-- ) > if( pthread_join( pthr[i], NULL ) != 0 ) > fprintf( stderr, "pthread_join\n" ); > > putchar( '.' ); > fflush( stdout ); > usleep( 25000 ); > } > } > > ----- snap ----- > > Compile them using -pthread and watch their ps output in another > window (FreeBSD-9.2 but that shouldn't matter). > > So what am I doing wrong here? > > Thanks, > > -Andre > _______________________________________________ > freebsd-threads at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to > "freebsd-threads-unsubscribe at freebsd.org" From mail at ma17.ata.myota.org Wed Feb 19 15:29:45 2014 From: mail at ma17.ata.myota.org (Andre Albsmeier) Date: Wed, 19 Feb 2014 16:29:32 +0100 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140219230824.0f2ba24b@X220.alogt.com> References: <20140218180646.GA67861@schlappy> <20140219144728.GA3036@schlappy> <20140219230824.0f2ba24b@X220.alogt.com> Message-ID: <20140219152932.GA3324@schlappy> On Wed, 19-Feb-2014 at 23:08:24 +0800, Erich Dollansky wrote: > Hi, > > as I understand your program, it creates 1000 threads, waits a but and > then starts again creating 1000 threads until something kill it. Right. 1000 threads are started, do nothing and thus terminate immediately. The value of 25 ms for usleep is just a good one for my machine to demonstrate the problem quickly while being sure that all 1000 threads have gone (verified with ps -H). You can chose a bigger value here and it will just take more time... > > From my point of view, your program depends very much on the default > settings of the environment. If the environment allows the immediate > execution of the new thread, you will never get many threads. > Otherwise, the number of threads hanging around could add up. It also > depends on the number of CPUs/cores your system has. But I never have more than 1000 threads. This is why I made pth2 where I explicitly wait for all 1000 threads to terminate before firing up the next bunch. So for me it seems as if every (already died away) thread leaves some memory behind which does not get free'ed or whatever. But it was my understanding that pthread_detach() or pthread_join() would clean all this up after a thread has finished. > > But your are right, it should not crash on a modern machine but it > still could use some amount of memory. But should it still grow with every bunch of 1000 threads being started while the previous 1000 are all gone already? I'd agree if the behaviour was like on the Linux machine where it seems that some kind of garbage collection kicks in for VIRT after a while but RES never goes above 1MB there. On FreeBSD it even makes the box swap... -Andre > > Erich > > On Wed, > 19 Feb 2014 15:47:28 +0100 Andre Albsmeier > wrote: > > > [Commenting my own mail from below and CC'ing freebsd-threads.] > > > > I have tested my code below on a Linux box (3.2.0) and here the > > behaviour is different and seems correct: While watching with > > top(1), VIRT climbs up to a few GB and collapses then to a few > > MB (somehow reminding me of some kind of garbage collection). > > Important thing is that RES always stays below 1MB. > > > > On FreeBSD SIZE and RES are about 100MB apart but both are > > increasing util 2GB and pth1 dies with > > > > Cannot map anonymous memory > > Out of memory > > > > So the question is: Is my programme buggy and Linux works around > > this bug or is there some kind of memory leak in the pthreads > > code in FreeBSD? > > > > -Andre > > > > ----- Forwarded message ----- > > > > Well, as these are my first steps regarding thread programming, > > it's probably my fault... > > > > Why does this programme slowly grow and grow until it hits > > resource limits? > > > > ----- snip pth1.c ----- > > > > #include > > #include > > #include > > > > void* mythread( void* arg ) > > { > > return NULL; > > } > > > > int main( int argc, const char* const argv[] ) > > { > > pthread_t pthr; > > int i; > > > > while( 1 ) { > > > > for( i=1000; i; i-- ) > > if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) > > fprintf( stderr, "pthread_create\n" ); > > else > > pthread_detach( pthr ); > > > > putchar( '.' ); > > fflush( stdout ); > > usleep( 25000 ); > > } > > } > > > > ----- snap ----- > > > > Just to be sure I have also created the non-detaching version > > which behaves in the same way: > > > > ----- snip pth2.c ----- > > > > #include > > #include > > #include > > > > #define M 1000 > > > > pthread_t pthr[M]; > > > > void* mythread( void* arg ) > > { > > return NULL; > > } > > > > int main( int argc, const char* const argv[] ) > > { > > int i; > > > > while( 1 ) { > > > > for( i=M; i; i-- ) > > if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 ) > > fprintf( stderr, "pthread_create\n" ); > > > > for( i=M; i; i-- ) > > if( pthread_join( pthr[i], NULL ) != 0 ) > > fprintf( stderr, "pthread_join\n" ); > > > > putchar( '.' ); > > fflush( stdout ); > > usleep( 25000 ); > > } > > } > > > > ----- snap ----- > > > > Compile them using -pthread and watch their ps output in another > > window (FreeBSD-9.2 but that shouldn't matter). > > > > So what am I doing wrong here? > > > > Thanks, > > > > -Andre > > _______________________________________________ > > freebsd-threads at freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > > To unsubscribe, send any mail to > > "freebsd-threads-unsubscribe at freebsd.org" > > -- echo "cdec cdec efg~ efg~ L8gagfL4ec L8gagfL4ec cc~ cc~" >/dev/speaker From bright at mu.org Wed Feb 19 15:30:57 2014 From: bright at mu.org (Alfred Perlstein) Date: Wed, 19 Feb 2014 07:30:52 -0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140219230824.0f2ba24b@X220.alogt.com> References: <20140218180646.GA67861@schlappy> <20140219144728.GA3036@schlappy> <20140219230824.0f2ba24b@X220.alogt.com> Message-ID: <5304CE2C.1060305@mu.org> On 2/19/14 7:08 AM, Erich Dollansky wrote: > Hi, > > as I understand your program, it creates 1000 threads, waits a but and > then starts again creating 1000 threads until something kill it. > > From my point of view, your program depends very much on the default > settings of the environment. If the environment allows the immediate > execution of the new thread, you will never get many threads. > Otherwise, the number of threads hanging around could add up. It also > depends on the number of CPUs/cores your system has. > > But your are right, it should not crash on a modern machine but it > still could use some amount of memory. OK. Maybe use a global locked with a mutex or atomic that gets increased each time the main thread spawns a thread and then decreased by that thread right before the child thread exits. Then the main loop can print the value of that. If it gets huge then your main loop is making threads faster than the system can remove them. If not , then it's a leak. > > Erich > > On Wed, > 19 Feb 2014 15:47:28 +0100 Andre Albsmeier > wrote: > >> [Commenting my own mail from below and CC'ing freebsd-threads.] >> >> I have tested my code below on a Linux box (3.2.0) and here the >> behaviour is different and seems correct: While watching with >> top(1), VIRT climbs up to a few GB and collapses then to a few >> MB (somehow reminding me of some kind of garbage collection). >> Important thing is that RES always stays below 1MB. >> >> On FreeBSD SIZE and RES are about 100MB apart but both are >> increasing util 2GB and pth1 dies with >> >> Cannot map anonymous memory >> Out of memory >> >> So the question is: Is my programme buggy and Linux works around >> this bug or is there some kind of memory leak in the pthreads >> code in FreeBSD? >> >> -Andre >> >> ----- Forwarded message ----- >> >> Well, as these are my first steps regarding thread programming, >> it's probably my fault... >> >> Why does this programme slowly grow and grow until it hits >> resource limits? >> >> ----- snip pth1.c ----- >> >> #include >> #include >> #include >> >> void* mythread( void* arg ) >> { >> return NULL; >> } >> >> int main( int argc, const char* const argv[] ) >> { >> pthread_t pthr; >> int i; >> >> while( 1 ) { >> >> for( i=1000; i; i-- ) >> if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) >> fprintf( stderr, "pthread_create\n" ); >> else >> pthread_detach( pthr ); >> >> putchar( '.' ); >> fflush( stdout ); >> usleep( 25000 ); >> } >> } >> >> ----- snap ----- >> >> Just to be sure I have also created the non-detaching version >> which behaves in the same way: >> >> ----- snip pth2.c ----- >> >> #include >> #include >> #include >> >> #define M 1000 >> >> pthread_t pthr[M]; >> >> void* mythread( void* arg ) >> { >> return NULL; >> } >> >> int main( int argc, const char* const argv[] ) >> { >> int i; >> >> while( 1 ) { >> >> for( i=M; i; i-- ) >> if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 ) >> fprintf( stderr, "pthread_create\n" ); >> >> for( i=M; i; i-- ) >> if( pthread_join( pthr[i], NULL ) != 0 ) >> fprintf( stderr, "pthread_join\n" ); >> >> putchar( '.' ); >> fflush( stdout ); >> usleep( 25000 ); >> } >> } >> >> ----- snap ----- >> >> Compile them using -pthread and watch their ps output in another >> window (FreeBSD-9.2 but that shouldn't matter). >> >> So what am I doing wrong here? >> >> Thanks, >> >> -Andre >> _______________________________________________ >> freebsd-threads at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-threads >> To unsubscribe, send any mail to >> "freebsd-threads-unsubscribe at freebsd.org" > _______________________________________________ > freebsd-threads at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe at freebsd.org" > -- Alfred Perlstein From emaste at freebsd.org Wed Feb 19 15:53:05 2014 From: emaste at freebsd.org (Ed Maste) Date: Wed, 19 Feb 2014 10:53:03 -0500 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140219230824.0f2ba24b@X220.alogt.com> References: <20140218180646.GA67861@schlappy> <20140219144728.GA3036@schlappy> <20140219230824.0f2ba24b@X220.alogt.com> Message-ID: On 19 February 2014 10:08, Erich Dollansky wrote: > Hi, > > as I understand your program, it creates 1000 threads, waits a but and > then starts again creating 1000 threads until something kill it. > > From my point of view, your program depends very much on the default > settings of the environment. If the environment allows the immediate > execution of the new thread, you will never get many threads. > Otherwise, the number of threads hanging around could add up. It also > depends on the number of CPUs/cores your system has. > But your are right, it should not crash on a modern machine but it > still could use some amount of memory. Andre's second example program demonstrates the problem as well. It calls pthread_join() for each of the created threads, so should not be able to leave them around. From mail at ma17.ata.myota.org Wed Feb 19 16:24:59 2014 From: mail at ma17.ata.myota.org (Andre Albsmeier) Date: Wed, 19 Feb 2014 17:24:41 +0100 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <5304CE2C.1060305@mu.org> References: <20140218180646.GA67861@schlappy> <20140219144728.GA3036@schlappy> <20140219230824.0f2ba24b@X220.alogt.com> <5304CE2C.1060305@mu.org> Message-ID: <20140219162441.GA3564@schlappy> On Wed, 19-Feb-2014 at 07:30:52 -0800, Alfred Perlstein wrote: > On 2/19/14 7:08 AM, Erich Dollansky wrote: > > Hi, > > > > as I understand your program, it creates 1000 threads, waits a but and > > then starts again creating 1000 threads until something kill it. > > > > From my point of view, your program depends very much on the default > > settings of the environment. If the environment allows the immediate > > execution of the new thread, you will never get many threads. > > Otherwise, the number of threads hanging around could add up. It also > > depends on the number of CPUs/cores your system has. > > > > But your are right, it should not crash on a modern machine but it > > still could use some amount of memory. > OK. Maybe use a global locked with a mutex or atomic that gets > increased each time the main thread spawns a thread and then decreased > by that thread right before the child thread exits. > Then the main loop can print the value of that. > > If it gets huge then your main loop is making threads faster than the > system can remove them. If not , then it's a leak. While my second programme (pth2) explicitly waits for all threads to terminate, I tried what you suggested and here is the result (although these are my first steps w.r.t. threads I think I did it correctly ;-)): ----- snip ----- #include #include #include #define THREADS 1000 int c = 0; pthread_mutex_t mtx = PTHREAD_MUTEX_INITIALIZER; void* mythread( void* arg ) { pthread_mutex_lock( &mtx ); c--; pthread_mutex_unlock( &mtx ); return NULL; } int main( int argc, const char* const argv[] ) { pthread_t pthr; int i; while( 1 ) { pthread_mutex_lock( &mtx ); c += THREADS; pthread_mutex_unlock( &mtx ); for( i=THREADS; i; i-- ) if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) fprintf( stderr, "pthread_create\n" ); else pthread_detach( pthr ); pthread_mutex_lock( &mtx ); fprintf( stderr, "%d:", c ); pthread_mutex_unlock( &mtx ); usleep( 25000 ); pthread_mutex_lock( &mtx ); fprintf( stderr, "%d ", c ); pthread_mutex_unlock( &mtx ); } } ----- snap ----- As we can see, I print c before the usleep() and after, both values separated by a colon. One of the results is here (after running it through fmt(1)): 1:0 0:0 1:0 0:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 0:0 1:0 1:0 1:0 1:0 1:0 322:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 126:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 16:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 131:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 66:0 1:0 98:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 132:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 18:0 1:0 1:0 1:0 1:0 1:0 1:0 123:0 1:0 1:0 1:0 1:0 1:0 1:0 24:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 146:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 121:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 3:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 125:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 52:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 7:0 1:0 1:0 1:0 2:0 1:0 573:0 1:0 1:0 1:0 1:0 1:0 1:0 465:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 19:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:0 1:^C We can see that in most case one thread is still active before the usleep() but after that all are gone. In rare cases there are hundreds active but they will be gone for sure after the usleep()... But apart from that we might stick to pth2.c as this definitely waits for all threads to terminate. I'd also be interested to know how pth2 behaves on FreeBSD boxes other than 9.2... -Andre > > > > > > Erich > > > > On Wed, > > 19 Feb 2014 15:47:28 +0100 Andre Albsmeier > > wrote: > > > >> [Commenting my own mail from below and CC'ing freebsd-threads.] > >> > >> I have tested my code below on a Linux box (3.2.0) and here the > >> behaviour is different and seems correct: While watching with > >> top(1), VIRT climbs up to a few GB and collapses then to a few > >> MB (somehow reminding me of some kind of garbage collection). > >> Important thing is that RES always stays below 1MB. > >> > >> On FreeBSD SIZE and RES are about 100MB apart but both are > >> increasing util 2GB and pth1 dies with > >> > >> Cannot map anonymous memory > >> Out of memory > >> > >> So the question is: Is my programme buggy and Linux works around > >> this bug or is there some kind of memory leak in the pthreads > >> code in FreeBSD? > >> > >> -Andre > >> > >> ----- Forwarded message ----- > >> > >> Well, as these are my first steps regarding thread programming, > >> it's probably my fault... > >> > >> Why does this programme slowly grow and grow until it hits > >> resource limits? > >> > >> ----- snip pth1.c ----- > >> > >> #include > >> #include > >> #include > >> > >> void* mythread( void* arg ) > >> { > >> return NULL; > >> } > >> > >> int main( int argc, const char* const argv[] ) > >> { > >> pthread_t pthr; > >> int i; > >> > >> while( 1 ) { > >> > >> for( i=1000; i; i-- ) > >> if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) > >> fprintf( stderr, "pthread_create\n" ); > >> else > >> pthread_detach( pthr ); > >> > >> putchar( '.' ); > >> fflush( stdout ); > >> usleep( 25000 ); > >> } > >> } > >> > >> ----- snap ----- > >> > >> Just to be sure I have also created the non-detaching version > >> which behaves in the same way: > >> > >> ----- snip pth2.c ----- > >> > >> #include > >> #include > >> #include > >> > >> #define M 1000 > >> > >> pthread_t pthr[M]; > >> > >> void* mythread( void* arg ) > >> { > >> return NULL; > >> } > >> > >> int main( int argc, const char* const argv[] ) > >> { > >> int i; > >> > >> while( 1 ) { > >> > >> for( i=M; i; i-- ) > >> if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 ) > >> fprintf( stderr, "pthread_create\n" ); > >> > >> for( i=M; i; i-- ) > >> if( pthread_join( pthr[i], NULL ) != 0 ) > >> fprintf( stderr, "pthread_join\n" ); > >> > >> putchar( '.' ); > >> fflush( stdout ); > >> usleep( 25000 ); > >> } > >> } > >> > >> ----- snap ----- > >> > >> Compile them using -pthread and watch their ps output in another > >> window (FreeBSD-9.2 but that shouldn't matter). > >> > >> So what am I doing wrong here? > >> > >> Thanks, > >> > >> -Andre > >> _______________________________________________ > >> freebsd-threads at freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-threads > >> To unsubscribe, send any mail to > >> "freebsd-threads-unsubscribe at freebsd.org" > > _______________________________________________ > > freebsd-threads at freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > > To unsubscribe, send any mail to "freebsd-threads-unsubscribe at freebsd.org" > > > > > -- > Alfred Perlstein > -- Stuxnet? Find ich gut. Manche lernen nur auf die harte Tour... From emaste at freebsd.org Wed Feb 19 16:52:47 2014 From: emaste at freebsd.org (Ed Maste) Date: Wed, 19 Feb 2014 11:52:46 -0500 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140219162441.GA3564@schlappy> References: <20140218180646.GA67861@schlappy> <20140219144728.GA3036@schlappy> <20140219230824.0f2ba24b@X220.alogt.com> <5304CE2C.1060305@mu.org> <20140219162441.GA3564@schlappy> Message-ID: On 19 February 2014 11:24, Andre Albsmeier wrote: > I'd also be interested to know how pth2 behaves on FreeBSD boxes > other than 9.2... It is reproducible on FreeBSD-CURRENT from about 2 weeks ago; it looks like we probably do have an issue in libthr. From georgekola at gmail.com Thu Feb 20 01:29:55 2014 From: georgekola at gmail.com (George Kola) Date: Wed, 19 Feb 2014 17:29:52 -0800 Subject: Question about ZFS arc cache and page cache Message-ID: I have a question about what happens when a file on a ZFS filesystem is mmaped into memory. I traced it on illumos and found that the file pages made it into the ARC cache and then I got a copy of it in page cache which was then mapped to my address space. Another process mapping the same file got the same page from the page cache. (I used mdb and converting virtual address to physical address to see if the pages were the same) . I was wondering if FreeBSD does the same copy or if the physical pages in ARC can be used for mmap without requiring a copy. The reason for this question is that we are currently on illumos and are considering moving to FreeBSD and are evaluating the pros and cons. Thanks, George From erichsfreebsdlist at alogt.com Thu Feb 20 03:39:20 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Thu, 20 Feb 2014 11:39:02 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140219162441.GA3564@schlappy> References: <20140218180646.GA67861@schlappy> <20140219144728.GA3036@schlappy> <20140219230824.0f2ba24b@X220.alogt.com> <5304CE2C.1060305@mu.org> <20140219162441.GA3564@schlappy> Message-ID: <20140220113902.63e97b00@X220.alogt.com> Hi, On Wed, 19 Feb 2014 17:24:41 +0100 Andre Albsmeier wrote: I am now running this program: > #include > #include > #include > > #define THREADS 1000 > > int c = 0; > pthread_mutex_t mtx = PTHREAD_MUTEX_INITIALIZER; > > void* mythread( void* arg ) > { > pthread_mutex_lock( &mtx ); > c--; > pthread_mutex_unlock( &mtx ); > > return NULL; > } > > int main( int argc, const char* const argv[] ) > { > pthread_t pthr; > int i; > > while( 1 ) { > > pthread_mutex_lock( &mtx ); > c += THREADS; > pthread_mutex_unlock( &mtx ); > > for( i=THREADS; i; i-- ) > if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) > fprintf( stderr, "pthread_create\n" ); > else > pthread_detach( pthr ); > > pthread_mutex_lock( &mtx ); > fprintf( stderr, "%d:", c ); > pthread_mutex_unlock( &mtx ); > > usleep( 25000 ); > > pthread_mutex_lock( &mtx ); > fprintf( stderr, "%d ", c ); > pthread_mutex_unlock( &mtx ); > } > } It runs on an i7 with two cores plus two hyper threads. The machine has 8GB of RAM and 16GB of Swap. The resident part of the program stays below 8GB (what wonder) but the rest is swapped out. I run this program on 10.0 from some 2 weeks ago. As already said, it looks like there stays some memory left after a thread terminates. To say something positive, the program really cleaned the memory. Erich From davidxu at freebsd.org Thu Feb 20 05:40:34 2014 From: davidxu at freebsd.org (David Xu) Date: Thu, 20 Feb 2014 13:41:08 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140218180646.GA67861@schlappy> References: <20140218180646.GA67861@schlappy> Message-ID: <53059574.8090605@freebsd.org> On 2014/02/19 02:06, Andre Albsmeier wrote: > Well, as these are my first steps regarding thread programming, > it's probably my fault... > > Why does this programme slowly grow and grow until it hits > resource limits? > > ----- snip pth1.c ----- > > #include > #include > #include > > void* mythread( void* arg ) > { > return NULL; > } > > int main( int argc, const char* const argv[] ) > { > pthread_t pthr; > int i; > > while( 1 ) { > > for( i=1000; i; i-- ) > if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 ) > fprintf( stderr, "pthread_create\n" ); > else > pthread_detach( pthr ); > > putchar( '.' ); > fflush( stdout ); > usleep( 25000 ); > } > } > > ----- snap ----- > > Just to be sure I have also created the non-detaching version > which behaves in the same way: > > ----- snip pth2.c ----- > > #include > #include > #include > > #define M 1000 > > pthread_t pthr[M]; > > void* mythread( void* arg ) > { > return NULL; > } > > int main( int argc, const char* const argv[] ) > { > int i; > > while( 1 ) { > > for( i=M; i; i-- ) > if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 ) > fprintf( stderr, "pthread_create\n" ); > > for( i=M; i; i-- ) > if( pthread_join( pthr[i], NULL ) != 0 ) > fprintf( stderr, "pthread_join\n" ); > > putchar( '.' ); > fflush( stdout ); > usleep( 25000 ); > } > } > > ----- snap ----- > > Compile them using -pthread and watch their ps output in another > window (FreeBSD-9.2 but that shouldn't matter). > > So what am I doing wrong here? > > Thanks, > > -Andre please compile it as static binary and run it, check if the problem still exists, I am hunting the bug, it is not necessary in the libthr because I have not changed its code for a long time. Regards, David Xu From erichsfreebsdlist at alogt.com Thu Feb 20 06:07:02 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Thu, 20 Feb 2014 14:06:44 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <53059574.8090605@freebsd.org> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> Message-ID: <20140220140644.7b1e0074@X220.alogt.com> Hi, On Thu, 20 Feb 2014 13:41:08 +0800 David Xu wrote: > On 2014/02/19 02:06, Andre Albsmeier wrote: > > please compile it as static binary and run it, check if the > problem still exists, I am hunting the bug, it is not necessary in > the libthr because I have not changed its code for a long time. I just compiled is a static program. The behaviour is now different. The size still grows but much slower while 'res' stays below some 10MB. Size also got stagnant after some 2 min CPU time hanging around 126MB. I am running it on: FreeBSD X220.alogt.com 10.0-STABLE FreeBSD 10.0-STABLE #15 r261342: Sat Feb 1 14:52:39 WITA 2014 erich at X220.alogt.com:/usr/obj/usr/src/sys/X220 amd64 Erich From davidxu at freebsd.org Thu Feb 20 08:05:56 2014 From: davidxu at freebsd.org (David Xu) Date: Thu, 20 Feb 2014 16:06:30 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140220140644.7b1e0074@X220.alogt.com> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> Message-ID: <5305B786.8020708@freebsd.org> On 2014/02/20 14:06, Erich Dollansky wrote: > Hi, > > On Thu, 20 Feb 2014 13:41:08 +0800 > David Xu wrote: > >> On 2014/02/19 02:06, Andre Albsmeier wrote: >> >> please compile it as static binary and run it, check if the >> problem still exists, I am hunting the bug, it is not necessary in >> the libthr because I have not changed its code for a long time. > > I just compiled is a static program. The behaviour is now different. > The size still grows but much slower while 'res' stays below some 10MB. > > Size also got stagnant after some 2 min CPU time hanging around 126MB. > > I am running it on: > > FreeBSD X220.alogt.com 10.0-STABLE FreeBSD 10.0-STABLE #15 r261342: Sat > Feb 1 14:52:39 WITA 2014 > erich at X220.alogt.com:/usr/obj/usr/src/sys/X220 amd64 > > Erich > I have found the bug, it is in rtld, where malloc_aligned() is misfunctioning, memory can be corrupted by the function. libthr calls _rtld_allocate_tls to allocate tls control block, the function is in rtld, its uses malloc_aligned() which is not working correctly. Patch is attached. Regards, David Xu -------------- next part -------------- Index: libexec/rtld-elf/xmalloc.c =================================================================== --- libexec/rtld-elf/xmalloc.c (revision 260700) +++ libexec/rtld-elf/xmalloc.c (working copy) @@ -72,14 +72,10 @@ malloc_aligned(size_t size, size_t align) { void *mem, *res; - uintptr_t x; - size_t asize, r; - r = round(sizeof(void *), align); - asize = round(size, align) + r; - mem = xmalloc(asize); - x = (uintptr_t)mem; - res = (void *)round(x, align); + mem = xmalloc(size + sizeof(void *) + align - 1); + res =(void*)(((uintptr_t)mem + sizeof(void *) + align - 1) & + ~(align - 1)); *(void **)((uintptr_t)res - sizeof(void *)) = mem; return (res); } From avg at FreeBSD.org Thu Feb 20 09:00:59 2014 From: avg at FreeBSD.org (Andriy Gapon) Date: Thu, 20 Feb 2014 10:59:48 +0200 Subject: Question about ZFS arc cache and page cache In-Reply-To: References: Message-ID: <5305C404.8010601@FreeBSD.org> on 20/02/2014 03:29 George Kola said the following: > I have a question about what happens when a file on a ZFS filesystem is mmaped into memory. I traced it on illumos and found that the file pages made it into the ARC cache and then I got a copy of it in page cache which was then mapped to my address space. Another process mapping the same file got the same page from the page cache. (I used mdb and converting virtual address to physical address to see if the pages were the same) . > I was wondering if FreeBSD does the same copy or if the physical pages in ARC can be used for mmap without requiring a copy. The reason for this question is that we are currently on illumos and are considering moving to FreeBSD and are evaluating the pros and cons. On FreeBSD the behavior is exactly the same in this respect. -- Andriy Gapon From mail at ma17.ata.myota.org Thu Feb 20 12:16:30 2014 From: mail at ma17.ata.myota.org (Andre Albsmeier) Date: Thu, 20 Feb 2014 13:16:05 +0100 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <5305B786.8020708@freebsd.org> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> Message-ID: <20140220121605.GA2156@schlappy> On Thu, 20-Feb-2014 at 16:06:30 +0800, David Xu wrote: > On 2014/02/20 14:06, Erich Dollansky wrote: > > Hi, > > > > On Thu, 20 Feb 2014 13:41:08 +0800 > > David Xu wrote: > > > >> On 2014/02/19 02:06, Andre Albsmeier wrote: > >> > >> please compile it as static binary and run it, check if the > >> problem still exists, I am hunting the bug, it is not necessary in > >> the libthr because I have not changed its code for a long time. > > > > I just compiled is a static program. The behaviour is now different. > > The size still grows but much slower while 'res' stays below some 10MB. > > > > Size also got stagnant after some 2 min CPU time hanging around 126MB. > > > > I am running it on: > > > > FreeBSD X220.alogt.com 10.0-STABLE FreeBSD 10.0-STABLE #15 r261342: Sat > > Feb 1 14:52:39 WITA 2014 > > erich at X220.alogt.com:/usr/obj/usr/src/sys/X220 amd64 > > > > Erich > > > > I have found the bug, it is in rtld, where malloc_aligned() is > misfunctioning, memory can be corrupted by the function. > > libthr calls _rtld_allocate_tls to allocate tls control block, > the function is in rtld, its uses malloc_aligned() which is not > working correctly. > > Patch is attached. Your patch seems to have fixed it here on 9.2-STABLE as well. Memory consumption stays at around 9MB RES and 130 MB SIZE for pth2. Thanks! -Andre > > Regards, > David Xu > > Index: libexec/rtld-elf/xmalloc.c > =================================================================== > --- libexec/rtld-elf/xmalloc.c (revision 260700) > +++ libexec/rtld-elf/xmalloc.c (working copy) > @@ -72,14 +72,10 @@ > malloc_aligned(size_t size, size_t align) > { > void *mem, *res; > - uintptr_t x; > - size_t asize, r; > > - r = round(sizeof(void *), align); > - asize = round(size, align) + r; > - mem = xmalloc(asize); > - x = (uintptr_t)mem; > - res = (void *)round(x, align); > + mem = xmalloc(size + sizeof(void *) + align - 1); > + res =(void*)(((uintptr_t)mem + sizeof(void *) + align - 1) & > + ~(align - 1)); > *(void **)((uintptr_t)res - sizeof(void *)) = mem; > return (res); > } -- My other computer is your windows box. From erichsfreebsdlist at alogt.com Fri Feb 21 04:44:30 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Fri, 21 Feb 2014 12:44:05 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <5305B786.8020708@freebsd.org> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> Message-ID: <20140221124405.0791d375@X220.alogt.com> Hi, On Thu, 20 Feb 2014 16:06:30 +0800 David Xu wrote: > On 2014/02/20 14:06, Erich Dollansky wrote: > > I have found the bug, it is in rtld, where malloc_aligned() is > misfunctioning, memory can be corrupted by the function. > > libthr calls _rtld_allocate_tls to allocate tls control block, > the function is in rtld, its uses malloc_aligned() which is not > working correctly. > I installed the patch. It is now much, much better but after hours, something still seems wrong. Size went above 200MB after 40min of CPU time. The number of threads is now above 1200. The machine also has 3 zombies. The machine was restarted some 4h ago. Erich From davidxu at freebsd.org Fri Feb 21 04:57:09 2014 From: davidxu at freebsd.org (David Xu) Date: Fri, 21 Feb 2014 12:57:44 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140221124405.0791d375@X220.alogt.com> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> <20140221124405.0791d375@X220.alogt.com> Message-ID: <5306DCC8.3080001@freebsd.org> On 2014/02/21 12:44, Erich Dollansky wrote: > Hi, > > On Thu, 20 Feb 2014 16:06:30 +0800 > David Xu wrote: > >> On 2014/02/20 14:06, Erich Dollansky wrote: >> >> I have found the bug, it is in rtld, where malloc_aligned() is >> misfunctioning, memory can be corrupted by the function. >> >> libthr calls _rtld_allocate_tls to allocate tls control block, >> the function is in rtld, its uses malloc_aligned() which is not >> working correctly. >> > I installed the patch. It is now much, much better but after hours, > something still seems wrong. Size went above 200MB after 40min of CPU > time. The number of threads is now above 1200. The machine also has 3 > zombies. The machine was restarted some 4h ago. > > Erich > Default thread stack size on 32-bit machine is 1M, if you have 1024 threads, the size can be larger than 1G. I checked maxinum threads kernel allowed is max_threads_per_proc which is 1500 default. So what you have seen might be normal. I don't know what is zombie, is it a zombie process? thread library does not create process. Regards, David Xu From erichsfreebsdlist at alogt.com Fri Feb 21 05:15:32 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Fri, 21 Feb 2014 13:15:17 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <5306DCC8.3080001@freebsd.org> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> <20140221124405.0791d375@X220.alogt.com> <5306DCC8.3080001@freebsd.org> Message-ID: <20140221131517.66d5b82c@X220.alogt.com> Hi, On Fri, 21 Feb 2014 12:57:44 +0800 David Xu wrote: > On 2014/02/21 12:44, Erich Dollansky wrote: > > On Thu, 20 Feb 2014 16:06:30 +0800 > > David Xu wrote: > > > >> On 2014/02/20 14:06, Erich Dollansky wrote: > >> > >> I have found the bug, it is in rtld, where malloc_aligned() is > >> misfunctioning, memory can be corrupted by the function. > >> > >> libthr calls _rtld_allocate_tls to allocate tls control block, > >> the function is in rtld, its uses malloc_aligned() which is not > >> working correctly. > >> > > I installed the patch. It is now much, much better but after hours, > > something still seems wrong. Size went above 200MB after 40min of > > CPU time. The number of threads is now above 1200. The machine also > > has 3 zombies. The machine was restarted some 4h ago. > > > > Default thread stack size on 32-bit machine is 1M, if you have > 1024 threads, the size can be larger than 1G. I checked > maxinum threads kernel allowed is max_threads_per_proc > which is 1500 default. So what you have seen might be normal. > I don't know what is zombie, is it a zombie process? > thread library does not create process. I know, the zombies do not fit the scenario. Irritating is that some threads stay over. Is it a side effect of the scheduler? I do my normal work with the machine. The load should not block this process for long as I do nothing which requires 100% for all the time. Should I add some code to wait until the counter is zero again and see what is happening then? Erich From davidxu at freebsd.org Fri Feb 21 05:40:58 2014 From: davidxu at freebsd.org (David Xu) Date: Fri, 21 Feb 2014 13:41:33 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140221131517.66d5b82c@X220.alogt.com> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> <20140221124405.0791d375@X220.alogt.com> <5306DCC8.3080001@freebsd.org> <20140221131517.66d5b82c@X220.alogt.com> Message-ID: <5306E70D.70402@freebsd.org> On 2014/02/21 13:15, Erich Dollansky wrote: > Hi, > > On Fri, 21 Feb 2014 12:57:44 +0800 > David Xu wrote: > >> On 2014/02/21 12:44, Erich Dollansky wrote: >>> On Thu, 20 Feb 2014 16:06:30 +0800 >>> David Xu wrote: >>> >>>> On 2014/02/20 14:06, Erich Dollansky wrote: >>>> >>>> I have found the bug, it is in rtld, where malloc_aligned() is >>>> misfunctioning, memory can be corrupted by the function. >>>> >>>> libthr calls _rtld_allocate_tls to allocate tls control block, >>>> the function is in rtld, its uses malloc_aligned() which is not >>>> working correctly. >>>> >>> I installed the patch. It is now much, much better but after hours, >>> something still seems wrong. Size went above 200MB after 40min of >>> CPU time. The number of threads is now above 1200. The machine also >>> has 3 zombies. The machine was restarted some 4h ago. >>> >> >> Default thread stack size on 32-bit machine is 1M, if you have >> 1024 threads, the size can be larger than 1G. I checked >> maxinum threads kernel allowed is max_threads_per_proc >> which is 1500 default. So what you have seen might be normal. >> I don't know what is zombie, is it a zombie process? >> thread library does not create process. > > I know, the zombies do not fit the scenario. Irritating is that some > threads stay over. Is it a side effect of the scheduler? I do my normal > work with the machine. The load should not block this process for long > as I do nothing which requires 100% for all the time. > I am running your code while playing flash video, and many threads are accumulated, sometimes I saw a peak number at 800 or more, I think it is normal, because the machine is loaded, after usleep returns, other newly created threads may still not have chance to run, and your next loop creates bunch of threads again, it could be very easily to exceed 1000 threads. > Should I add some code to wait until the counter is zero again and see > what is happening then? > Yes, I think so. But thread stacks will be cached by libthr, and never return to free space. so even if there is only one thread, you may see a very large size in top. > Erich > From erichsfreebsdlist at alogt.com Fri Feb 21 06:05:55 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Fri, 21 Feb 2014 14:05:41 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <5306E70D.70402@freebsd.org> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> <20140221124405.0791d375@X220.alogt.com> <5306DCC8.3080001@freebsd.org> <20140221131517.66d5b82c@X220.alogt.com> <5306E70D.70402@freebsd.org> Message-ID: <20140221140541.72020e93@X220.alogt.com> Hi, On Fri, 21 Feb 2014 13:41:33 +0800 David Xu wrote: > On 2014/02/21 13:15, Erich Dollansky wrote: > > On Fri, 21 Feb 2014 12:57:44 +0800 > > David Xu wrote: > > > I am running your code while playing flash video, and many threads are it is not even my code. I am just interested. > accumulated, sometimes I saw a peak number at 800 or more, I think > it is normal, because the machine is loaded, after usleep returns, > other newly created threads may still not have chance to run, and > your next loop creates bunch of threads again, it could be very > easily to exceed 1000 threads. > > > Should I add some code to wait until the counter is zero again and > > see what is happening then? > > > > Yes, I think so. But thread stacks will be cached by libthr, and never > return to free space. so even if there is only one thread, you may > see a very large size in top. > I first called usleep only of more than 10 threads remained active. Top showed then nicely that this happens sometimes. Since I call usleep only when more than 1000 threads remain active, looks even funnier on top. I added then a loop to the thread and it works smoothly now. It runs now also into problems creating new threads when too many are still running. Of course, the program takes now much more CPU time as it waits less. I think that we can say that the library is not really optimised for empty threads. So, all seems perfect now. Thanks for your work. Erich From ivoras at freebsd.org Fri Feb 21 11:53:09 2014 From: ivoras at freebsd.org (Ivan Voras) Date: Fri, 21 Feb 2014 12:52:48 +0100 Subject: ata.4 changes Message-ID: Hello, Can someone who knows more about man pages take a look at this patch to ata.4: Index: ada.4 =================================================================== --- ada.4 (revision 262293) +++ ada.4 (working copy) @@ -25,7 +25,7 @@ .\" .\" $FreeBSD$ .\" -.Dd February 8, 2012 +.Dd February 21, 2014 .Dt ADA 4 .Os .Sh NAME @@ -129,8 +129,14 @@ These variables determines whether device write cache should be enabled globally or per-device or disabled. Set to 1 to enable write cache, 0 to disable, -1 to leave it as-is. -Values modified in runtime take effect only after device reset. -The global default is currently enabled. +Values modified in runtime take effect only after device reset (using +.Xr camcontrol 8 +reset). +Because of that, this setting should be changed in a +.Xr loader 8 +tunable, not in +.Pa /etc/sysctl.conf . +The global default is currently 1. The per-device default is to leave it as-is (follow global setting). .El .Sh FILES - is it according to style and am I using the tags correctly? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 391 bytes Desc: OpenPGP digital signature URL: From hrs at FreeBSD.org Fri Feb 21 12:11:46 2014 From: hrs at FreeBSD.org (Hiroki Sato) Date: Fri, 21 Feb 2014 21:10:26 +0900 (JST) Subject: ata.4 changes In-Reply-To: References: Message-ID: <20140221.211026.115752007914804800.hrs@allbsd.org> Ivan Voras wrote in : iv> Hello, iv> iv> Can someone who knows more about man pages take a look at this patch to iv> ata.4: iv> iv> Index: ada.4 iv> =================================================================== iv> --- ada.4 (revision 262293) iv> +++ ada.4 (working copy) iv> @@ -25,7 +25,7 @@ iv> .\" iv> .\" $FreeBSD$ iv> .\" iv> -.Dd February 8, 2012 iv> +.Dd February 21, 2014 iv> .Dt ADA 4 iv> .Os iv> .Sh NAME iv> @@ -129,8 +129,14 @@ iv> These variables determines whether device write cache should be enabled iv> globally or per-device or disabled. iv> Set to 1 to enable write cache, 0 to disable, -1 to leave it as-is. iv> -Values modified in runtime take effect only after device reset. iv> -The global default is currently enabled. iv> +Values modified in runtime take effect only after device reset (using iv> +.Xr camcontrol 8 iv> +reset). .Pq should be used for (): Values modified in runtime take effect only after device reset .Pq using reset subcommand of Xr camcontrol 8 . iv> +Because of that, this setting should be changed in a iv> +.Xr loader 8 iv> +tunable, not in iv> +.Pa /etc/sysctl.conf . I think /boot/loader.conf is easier to understand: Because of that, this setting should be changed in .Pa /boot/loader.conf , not in .Pa /etc/sysctl.conf . iv> +The global default is currently 1. iv> The per-device default is to leave it as-is (follow global setting). iv> .El iv> .Sh FILES iv> iv> - is it according to style and am I using the tags correctly? -- Hiroki -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: not available URL: From andre at albsmeier.net Fri Feb 21 12:21:48 2014 From: andre at albsmeier.net (Andre Albsmeier) Date: Fri, 21 Feb 2014 13:21:22 +0100 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <5306E70D.70402@freebsd.org> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> <20140221124405.0791d375@X220.alogt.com> <5306DCC8.3080001@freebsd.org> <20140221131517.66d5b82c@X220.alogt.com> <5306E70D.70402@freebsd.org> Message-ID: <20140221122122.GA2031@schlappy> On Fri, 21-Feb-2014 at 13:41:33 +0800, David Xu wrote: > On 2014/02/21 13:15, Erich Dollansky wrote: > > Hi, > > > > On Fri, 21 Feb 2014 12:57:44 +0800 > > David Xu wrote: > > > >> On 2014/02/21 12:44, Erich Dollansky wrote: > >>> On Thu, 20 Feb 2014 16:06:30 +0800 > >>> David Xu wrote: > >>> > >>>> On 2014/02/20 14:06, Erich Dollansky wrote: > >>>> > >>>> I have found the bug, it is in rtld, where malloc_aligned() is > >>>> misfunctioning, memory can be corrupted by the function. > >>>> > >>>> libthr calls _rtld_allocate_tls to allocate tls control block, > >>>> the function is in rtld, its uses malloc_aligned() which is not > >>>> working correctly. > >>>> > >>> I installed the patch. It is now much, much better but after hours, > >>> something still seems wrong. Size went above 200MB after 40min of > >>> CPU time. The number of threads is now above 1200. The machine also > >>> has 3 zombies. The machine was restarted some 4h ago. > >>> > >> > >> Default thread stack size on 32-bit machine is 1M, if you have > >> 1024 threads, the size can be larger than 1G. I checked > >> maxinum threads kernel allowed is max_threads_per_proc > >> which is 1500 default. So what you have seen might be normal. > >> I don't know what is zombie, is it a zombie process? > >> thread library does not create process. > > > > I know, the zombies do not fit the scenario. Irritating is that some > > threads stay over. Is it a side effect of the scheduler? I do my normal > > work with the machine. The load should not block this process for long > > as I do nothing which requires 100% for all the time. > > > > I am running your code while playing flash video, and many threads are > accumulated, sometimes I saw a peak number at 800 or more, I think > it is normal, because the machine is loaded, after usleep returns, > other newly created threads may still not have chance to run, and > your next loop creates bunch of threads again, it could be very > easily to exceed 1000 threads. I would like to recall that my usleep programme was just a first shot to demonstrate the problem here on my idle machine. For real work loads the other one (pth2) might be better as this one waits for all threads to terminate before firing up new ones... -Andre > > > Should I add some code to wait until the counter is zero again and see > > what is happening then? > > > > Yes, I think so. But thread stacks will be cached by libthr, and never > return to free space. so even if there is only one thread, you may > see a very large size in top. > > > Erich > > > -- Micro$oft: Which virus will you get today? From erichsfreebsdlist at alogt.com Fri Feb 21 14:17:42 2014 From: erichsfreebsdlist at alogt.com (Erich Dollansky) Date: Fri, 21 Feb 2014 22:17:28 +0800 Subject: pthread programming eats up resources (My or FreeBSD's fault?) In-Reply-To: <20140221122122.GA2031@schlappy> References: <20140218180646.GA67861@schlappy> <53059574.8090605@freebsd.org> <20140220140644.7b1e0074@X220.alogt.com> <5305B786.8020708@freebsd.org> <20140221124405.0791d375@X220.alogt.com> <5306DCC8.3080001@freebsd.org> <20140221131517.66d5b82c@X220.alogt.com> <5306E70D.70402@freebsd.org> <20140221122122.GA2031@schlappy> Message-ID: <20140221221728.5464e035@X220.alogt.com> Hi, On Fri, 21 Feb 2014 13:21:22 +0100 Andre Albsmeier wrote: > On Fri, 21-Feb-2014 at 13:41:33 +0800, David Xu wrote: > > On 2014/02/21 13:15, Erich Dollansky wrote: > I would like to recall that my usleep programme was just a first > shot to demonstrate the problem here on my idle machine. For real > work loads the other one (pth2) might be better as this one waits > for all threads to terminate before firing up new ones... do not worry. It was the perfect thing to demonstrate the problem. We all should be happy for this masterpiece of code. Erich From wblock at wonkity.com Fri Feb 21 14:51:24 2014 From: wblock at wonkity.com (Warren Block) Date: Fri, 21 Feb 2014 07:51:15 -0700 (MST) Subject: ata.4 changes In-Reply-To: <20140221.211026.115752007914804800.hrs@allbsd.org> References: <20140221.211026.115752007914804800.hrs@allbsd.org> Message-ID: On Fri, 21 Feb 2014, Hiroki Sato wrote: > Ivan Voras wrote > in : > > iv> Hello, > iv> > iv> Can someone who knows more about man pages take a look at this patch to > iv> ata.4: > iv> > iv> Index: ada.4 > iv> =================================================================== > iv> --- ada.4 (revision 262293) > iv> +++ ada.4 (working copy) > iv> @@ -25,7 +25,7 @@ > iv> .\" > iv> .\" $FreeBSD$ > iv> .\" > iv> -.Dd February 8, 2012 > iv> +.Dd February 21, 2014 > iv> .Dt ADA 4 > iv> .Os > iv> .Sh NAME > iv> @@ -129,8 +129,14 @@ > iv> These variables determines whether device write cache should be enabled Not part of the original change, but since we are here: s/determines/determine/ > iv> globally or per-device or disabled. > iv> Set to 1 to enable write cache, 0 to disable, -1 to leave it as-is. > iv> -Values modified in runtime take effect only after device reset. > iv> -The global default is currently enabled. > iv> +Values modified in runtime take effect only after device reset (using > iv> +.Xr camcontrol 8 > iv> +reset). > > .Pq should be used for (): > > Values modified in runtime take effect only after device reset > .Pq using reset subcommand of Xr camcontrol 8 . > > iv> +Because of that, this setting should be changed in a > iv> +.Xr loader 8 > iv> +tunable, not in > iv> +.Pa /etc/sysctl.conf . > > I think /boot/loader.conf is easier to understand: > > Because of that, this setting should be changed in > .Pa /boot/loader.conf , > not in > .Pa /etc/sysctl.conf . Agreed. From ian at FreeBSD.org Fri Feb 21 17:24:29 2014 From: ian at FreeBSD.org (Ian Lepore) Date: Fri, 21 Feb 2014 10:24:25 -0700 Subject: What is the precision arg for callout_reset_sbt() and friends? Message-ID: <1393003465.1145.111.camel@revolution.hippie.lan> I can't figure out from the code or the manpage what should be passed as the precision argument to the sleep-related functions that take sbintime values. It seems like most existing code passes 0, what effect does that have? What should I pass if precision really matters? What if it doesn't matter at all? Most of the time I want something like a 1ms timeout where it really isn't critical if it turns into 2 or even 5ms. Other times when I say 1ms I really want something close to 1ms. -- Ian From rizzo at iet.unipi.it Fri Feb 21 20:24:28 2014 From: rizzo at iet.unipi.it (Luigi Rizzo) Date: Fri, 21 Feb 2014 21:26:37 +0100 Subject: What is the precision arg for callout_reset_sbt() and friends? In-Reply-To: <1393003465.1145.111.camel@revolution.hippie.lan> References: <1393003465.1145.111.camel@revolution.hippie.lan> Message-ID: <20140221202637.GC5118@onelab2.iet.unipi.it> On Fri, Feb 21, 2014 at 10:24:25AM -0700, Ian Lepore wrote: > I can't figure out from the code or the manpage what should be passed as > the precision argument to the sleep-related functions that take sbintime > values. It seems like most existing code passes 0, what effect does > that have? What should I pass if precision really matters? What if it > doesn't matter at all? I was one of the people suggesting the addition of a 'precision' argument, the intent being to use it as a hint to help merge events and reduce the number of timer interrupts. Since pre-existing APIs did not have this parameter we used 0 when translating from the old to the new one. If you write something new of course you can express your needs as you suggest below. In practice i think the following issues partly reduce the usefulness of the parameter: - there is no way to express a precision from userspace so anything using select/poll/kqueue will end up with the 'standard' argument; - with large intervals, aggregation is not really a concern: events are likely to be scattered (so little chance of merging) and probably infrequent (so little gain from merging) - with short periodic intervals (where merging might save something), applications would prefer if the interval is reasonably accurate and has no drift. So I would expect to see a lot of precision=0 in clients of this type. - the implementation, for a delay D and precision P, seems to schedule a timeout after D+P unless one exists already in the interval D..D+P. This increases the chance of merging but is annoying for applications. I think we should do it differently i.e.: if nothing is scheduled in D..D+P then schedule a timeout after D (and not D+P). Then, in the data structure we should (hopefully it is already like this) keep both D and P so when we grab the next event from the timeout queue, we try to set the timer to a value that includes the largest number of intervals, using one of the D_i values (so that at least someone will get it close to the request). This might require a slightly different data structure than the timing wheel/heap or whatever is used now as a backend. cheers luigi > Most of the time I want something like a 1ms timeout where it really > isn't critical if it turns into 2 or even 5ms. Other times when I say > 1ms I really want something close to 1ms. > > -- Ian > > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" From julian at freebsd.org Sat Feb 22 02:34:02 2014 From: julian at freebsd.org (Julian Elischer) Date: Sat, 22 Feb 2014 10:21:19 +0800 Subject: BSD XXI Manifesto [agree] [intersting] In-Reply-To: <20140218072821.GF34282@FreeBSD.org> References: <20140218072821.GF34282@FreeBSD.org> Message-ID: <5308099F.4090706@freebsd.org> On 2/18/14, 3:28 PM, Wojciech A. Koszek wrote: > (cross-posted message: eventual discussion let's keep on hackers@) > > Hello, > > After being disappointed with the list of submitted FreeBSD ideas, I created > my own Machiavellist vision of XXI-century FreeBSD. I paste it below. If you > want to add something, it's here: > > https://wiki.freebsd.org/BSD_XXI_Manifesto > > GSOC students could use this as an inspiration for their projects. The idea > is to invite non-C, non-OS, non-kernel developers to help out with FreeBSD > stuff. > > ============ > > BSDXXI manifesto [nice stuff] removed for brevity I like all this.. I thought you meant XXI to mean the "FreeBSD's 21st year" but there is more than one year's worth of stuff there. I really suggest people seriously look at the list.. lots of really neat ideas. peole who are not necessarily C coders could do lots of this if we had a project to gather people under to do it. PCBSD people would be a core of interested people.. From wjw at digiware.nl Sat Feb 22 23:06:53 2014 From: wjw at digiware.nl (Willem Jan Withagen) Date: Sun, 23 Feb 2014 00:06:43 +0100 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> Message-ID: <53092D83.6050603@digiware.nl> On 16-2-2014 6:16, Perry Hutchison wrote: > Jordan Hubbard wrote: > >> Even variant symlinks (/bin -> /${ARCH}/bin), which can expand >> differently depending on the user context, have clearly >> understandable semantics - you know that the symlink is going >> to expand to exactly one file no matter what ARCH is set to. > > s/file/pathname/ > > Depending on what ARCH is set to, the expanision may or may not > point to any actual file (or directory, or ...) Yes, please can we get these .... Apollo Domain systems had those, and they were great. Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or SYSV to get the other stuff. Would indeed work great for things like /bin or even /usr/local/etc -> /${HOST}/usr/local/etc I was running a special patch version 2.2 at one time, that had variant replacement in the lookup-cache routines. But I never was able to figure out a handy way to get the info back into the kernel. So I gave up. One would need to get at the user environment of the process, and then parse and scrutinize the ENV every time you need to use a replacement. So probably libc is the place to put it, but then you get into trouble again when somebody uses the not standard libc... Also got a lot of flack from people suggesting it would create security problems.... (I beg to differ) But I would really like the timewarp back to 1985. --WjW From jordan.hubbard at gmail.com Sun Feb 23 02:45:29 2014 From: jordan.hubbard at gmail.com (Jordan Hubbard) Date: Sat, 22 Feb 2014 18:45:25 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <53092D83.6050603@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> Message-ID: <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen wrote: > Yes, please can we get these .... > > Apollo Domain systems had those, and they were great. > Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or > SYSV to get the other stuff. Yep, I loved these things on Domain/OS! We system admin types used them to do all kinds of clever (and useful) things. Looks like FreeBSD has actually *had* an implementation for 6 years now. I don?t necessarily agree with the architectural decision to create a different namespace and command (varsym) to manipulate it - it was really nice just having it be a part of the standard environ(7) - but hey, any implementation is better than no implementation. Whatever happened to https://wiki.freebsd.org/200808DevSummit?action=AttachFile&do=get&target=variant-symlinks-for-freebsd.pdf ? - Jordan From jordan.hubbard at gmail.com Sun Feb 23 03:13:14 2014 From: jordan.hubbard at gmail.com (Jordan Hubbard) Date: Sat, 22 Feb 2014 19:13:09 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <53092D83.6050603@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> Message-ID: [ Whoops, my previous reply did not do this one justice - replying to the other parts of the message now! ] On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen wrote: > I was running a special patch version 2.2 at one time, that had variant > replacement in the lookup-cache routines. But I never was able to figure > out a handy way to get the info back into the kernel. IIRC, Domain/OS dealt with this by making the environment part of the kernel namespace - e.g. the userland access to it, getenv/setenv/**env, were actually fault requests back into the kernel for the information. I believe there were even some limitations on setenv replacing existing entries because of that, but I may be mis-remembering another Unix?s recapitulation of the same feature. Domain/OS was very influential in that respect, and I recall several others who tried the variant symlink idea, or riffed off the concept in a more limited fashion (Pyramid?s ?universe? concept and conditional symbolic links comes to mind). I think to do it now, however, we could take an approach which you mention but I think don?t credit enough: > So I gave up. One would need to get at the user environment of the process, and then parse > and scrutinize the ENV every time you need to use a replacement. So > probably libc is the place to put it, but then you get into trouble > again when somebody uses the not standard libc? Which is interposing at the libc level? I?ve seen a number of surprisingly successful ?science experiments? with filesystem interposing at the libc boundary, trapping all the filesystem access functions in order to, for example, figure out during a software package?s build process, just what header files it read, which helper binaries it executed, and essentially what its dependencies on other software packages were. Same with variant symlinks, or per-process namespaces (where a given process tree sees a different view of the filesystem hierarchy than others). I think there are so few static binaries on your average Unix box now, at least not that aren?t completely relegated to various ?rescue? roles where such features are less than interesting anyway, that doing it at the libc level is an entirely reasonable approach for the 99% case. > Also got a lot of flack from people suggesting it would create security problems.... (I beg to differ) I would also beg to differ. If you don?t have access to the target of a symlink without variable expansion, I don?t see how access *with* variable expansion is going to make one damn bit of difference! The security crowd was probably more concerned with the notions of deliberately overflowing the expansion buffer by creating environment variables that expanded hugely, or to things with weird special characters embedded in them, but both of those challenges are easy to overcome. If you choose to do it in userland, it?s even easier. Either way, I think it?s fair to say that one can shoot down almost any good or interesting idea with ?but but it could create a security hole!!" and I think that does the cause of evolution much disservice. Some OS folks like to think that their corner of the universe is where the rubber really meets the road in terms of (protecting, providing, enforcing) security, but that?s just hubris coupled with a longing for the ?good old days? when such fallacies of thought was closer to being true. Now, of course, the bulk security attacks are centered around tricking users into doing things that compromise their own security, much like a good confidence trickster can con someone out of large sums of money regardless of the fact that there?s a security guard with a gun at their bank. The guard can?t do much if a customer walks in and withdraws their own money to give to the con man who?s waiting outside the front entrance. :-) But, ahem, I digress!! > But I would really like the timewarp back to 1985. Indeed. I often tell interns who are looking for interesting project ideas to simply look back into our own past. Almost all the really interesting and cool research activities where operating systems are concerned seems to have happened between the years of 1970-1990. Sprite, Plan 9, Mach (hey, file space name servers anyone?), Domain OS, all kinds of neat ideas that sadly died or were forgotten in the name of consolidation, performance and expedience. Indeed, the performance of some of those concepts was actually rather woeful when 4MB of memory and 1MIP were all you one to work with. Maybe now that we have more hardware horsepower than we almost know what to do with, it?s time for some of those ideas to enjoy a renaissance? Sounds like a good EuroBSDCon or BSDCan talk. :-) - Jordan From wjw at digiware.nl Sun Feb 23 11:53:40 2014 From: wjw at digiware.nl (Willem Jan Withagen) Date: Sun, 23 Feb 2014 12:53:38 +0100 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> Message-ID: <5309E142.2020504@digiware.nl> On 23-2-2014 4:13, Jordan Hubbard wrote: > [ Whoops, my previous reply did not do this one justice - replying to > the other parts of the message now! ] > > On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen > wrote: > >> I was running a special patch version 2.2 at one time, that had >> variant replacement in the lookup-cache routines. But I never was >> able to figure out a handy way to get the info back into the >> kernel. > > IIRC, Domain/OS dealt with this by making the environment part of the > kernel namespace - e.g. the userland access to it, > getenv/setenv/**env, were actually fault requests back into the > kernel for the information. I believe there were even some > limitations on setenv replacing existing entries because of that, but > I may be mis-remembering another Unix?s recapitulation of the same > feature. Domain/OS was very influential in that respect, and I > recall several others who tried the variant symlink idea, or riffed > off the concept in a more limited fashion (Pyramid?s ?universe? > concept and conditional symbolic links comes to mind). One was not able to replace all ENV variables in such a way that it made it into the kernel for variant replacement. Things like NodeID etc... were hardwired. But I don't remember any serious limitations that hindered Unix type behaviour. I probably still have some Domain/OS architecture whitepapers describing hthis. But you are probably right in that it was actually part of the users kernel-space. [ I remember that the way Domain/OS translated its internal PID into the Unix PID was much more cumbersome. The details have evaporated in history, other than that I had to give Wietse Venema access to our boxes to actually get him convinced that I did not smoke anything when doing some tests for him.] > I think to do it now, however, we could take an approach which you > mention but I think don?t credit enough: > >> So I gave up. One would need to get at the user environment of the >> process, and then parse and scrutinize the ENV every time you need >> to use a replacement. So probably libc is the place to put it, but >> then you get into trouble again when somebody uses the not standard >> libc? > > Which is interposing at the libc level? I?ve seen a number of > surprisingly successful ?science experiments? with filesystem > interposing at the libc boundary, trapping all the filesystem access > functions in order to, for example, figure out during a software > package?s build process, just what header files it read, which helper > binaries it executed, and essentially what its dependencies on other > software packages were. Same with variant symlinks, or per-process > namespaces (where a given process tree sees a different view of the > filesystem hierarchy than others). I think there are so few static > binaries on your average Unix box now, at least not that aren?t > completely relegated to various ?rescue? roles where such features > are less than interesting anyway, that doing it at the libc level is > an entirely reasonable approach for the 99% case. > >> Also got a lot of flack from people suggesting it would create >> security problems.... (I beg to differ) > > I would also beg to differ. If you don?t have access to the target > of a symlink without variable expansion, I don?t see how access > *with* variable expansion is going to make one damn bit of > difference! The security crowd was probably more concerned with the > notions of deliberately overflowing the expansion buffer by creating > environment variables that expanded hugely, or to things with weird > special characters embedded in them, but both of those challenges are > easy to overcome. If you choose to do it in userland, it?s even > easier. Which was exactly my argument. But it could create obfuscation and sorts. I put it more to "what we don't know, we don't like" attitude. And like I said, the business I was running took off, the patches succumbed to bit-rot. I especially like the solution of putting it in kernel-space, which allows for one-time careful scanning... Perhaps even have a setenv/export -vl to push env-values into a special VL-env-buffer? and have a syscall(VLbuffer, {set,get,del}, name, val) With help I would be tempted to dig into this again... Haven't done any serious programming for quite some time. >> But I would really like the timewarp back to 1985. > > Indeed. I often tell interns who are looking for interesting project > ideas to simply look back into our own past. Almost all the really > interesting and cool research activities where operating systems are > concerned seems to have happened between the years of 1970-1990. > Sprite, Plan 9, Mach (hey, file space name servers anyone?), Domain > OS, all kinds of neat ideas that sadly died or were forgotten in the > name of consolidation, performance and expedience. Indeed, the > performance of some of those concepts was actually rather woeful when > 4MB of memory and 1MIP were all you one to work with. Maybe now that > we have more hardware horsepower than we almost know what to do with, > it?s time for some of those ideas to enjoy a renaissance? Sounds > like a good EuroBSDCon or BSDCan talk. :-) Well I do regular talks about the state of the internet. And I start by showing people a picture of an acoustic 300/300 baud modem (1980). to compare that to 500Mbit fiber consumer links that you can get here in the Netherlands. Some hold for CPU speed, or memory space. And then we look at the growth of capabilities of applications, and wonder if we do see a 10^6 growth there too. And I can't help the feeling that we should be able to do more with what we got. But then talking like this, makes me understand why my hairs have gone grey. :) --WjW From wjw at digiware.nl Sun Feb 23 12:49:09 2014 From: wjw at digiware.nl (Willem Jan Withagen) Date: Sun, 23 Feb 2014 13:49:09 +0100 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> Message-ID: <5309EE45.8090907@digiware.nl> On 23-2-2014 3:45, Jordan Hubbard wrote: > > On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen > wrote: > >> Yes, please can we get these .... >> >> Apollo Domain systems had those, and they were great. Set SYSTYPE >> to BSD4 and get the BSD tree and all that came with it, or SYSV to >> get the other stuff. > > Yep, I loved these things on Domain/OS! We system admin types used > them to do all kinds of clever (and useful) things. > > Looks like FreeBSD has actually *had* an implementation for 6 years > now. I don?t necessarily agree with the architectural decision to > create a different namespace and command (varsym) to manipulate it - > it was really nice just having it be a part of the standard > environ(7) - but hey, any implementation is better than no > implementation. Whatever happened to > https://wiki.freebsd.org/200808DevSummit?action=AttachFile&do=get&target=variant-symlinks-for-freebsd.pdf This one came in after your previous one, which I answered extensively. But I see your point in just using ENV, makes life much more consistent. And then again: Any version is better than no version. Could do integration in due time. --WjW From deischen at freebsd.org Sun Feb 23 15:32:18 2014 From: deischen at freebsd.org (Daniel Eischen) Date: Sun, 23 Feb 2014 10:18:31 -0500 (EST) Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <53092D83.6050603@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> Message-ID: On Sun, 23 Feb 2014, Willem Jan Withagen wrote: > On 16-2-2014 6:16, Perry Hutchison wrote: >> Jordan Hubbard wrote: >> >>> Even variant symlinks (/bin -> /${ARCH}/bin), which can expand >>> differently depending on the user context, have clearly >>> understandable semantics - you know that the symlink is going >>> to expand to exactly one file no matter what ARCH is set to. >> >> s/file/pathname/ >> >> Depending on what ARCH is set to, the expanision may or may not >> point to any actual file (or directory, or ...) > > Yes, please can we get these .... > > Apollo Domain systems had those, and they were great. > Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or > SYSV to get the other stuff. > > Would indeed work great for things like /bin or even > /usr/local/etc -> /${HOST}/usr/local/etc This topic comes up every couple of years. I recall Domain OS fondly - it was my first UNIX-like OS. I would really like variant symlinks, but I predict in another couple of years we'll be having the same conversation :-) -- DE From wjw at digiware.nl Sun Feb 23 16:13:47 2014 From: wjw at digiware.nl (Willem Jan Withagen) Date: Sun, 23 Feb 2014 17:13:39 +0100 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> Message-ID: <530A1E33.2060007@digiware.nl> On 23-2-2014 16:18, Daniel Eischen wrote: > This topic comes up every couple of years. I recall > Domain OS fondly - it was my first UNIX-like OS. I would > really like variant symlinks, but I predict in another > couple of years we'll be having the same conversation :-) I did have some exposure with cmd-line unix Sys3, but only after Domain/OS I understood what power was there. Usually when VariantLinks come up, I try to light the fire.... First time was round 2.x, and a few more times in between. And as most say: Show us the code. But Jordan found evidence that is was almost already there. And even then it did not get through. So I'm more than reluctant to burn the few spare hours I have, if it is rejected upon forehand. --WjW From gljennjohn at gmail.com Sun Feb 23 16:30:48 2014 From: gljennjohn at gmail.com (Gary Jennejohn) Date: Sun, 23 Feb 2014 17:30:42 +0100 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> Message-ID: <20140223173042.074d3eb0@ernst.home> On Sun, 23 Feb 2014 10:18:31 -0500 (EST) Daniel Eischen wrote: > On Sun, 23 Feb 2014, Willem Jan Withagen wrote: > > > On 16-2-2014 6:16, Perry Hutchison wrote: > >> Jordan Hubbard wrote: > >> > >>> Even variant symlinks (/bin -> /${ARCH}/bin), which can expand > >>> differently depending on the user context, have clearly > >>> understandable semantics - you know that the symlink is going > >>> to expand to exactly one file no matter what ARCH is set to. > >> > >> s/file/pathname/ > >> > >> Depending on what ARCH is set to, the expanision may or may not > >> point to any actual file (or directory, or ...) > > > > Yes, please can we get these .... > > > > Apollo Domain systems had those, and they were great. > > Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or > > SYSV to get the other stuff. > > > > Would indeed work great for things like /bin or even > > /usr/local/etc -> /${HOST}/usr/local/etc > > This topic comes up every couple of years. I recall > Domain OS fondly - it was my first UNIX-like OS. I would > really like variant symlinks, but I predict in another > couple of years we'll be having the same conversation :-) > Hear, hear! When I saw the first post I immediately thought "is it 1994 again?" Well, maybe the first discussion wasn't in 1994, but it was quite some time ago. -- Gary Jennejohn From wjw at digiware.nl Sun Feb 23 19:10:59 2014 From: wjw at digiware.nl (Willem Jan Withagen) Date: Sun, 23 Feb 2014 20:10:53 +0100 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <20140223173042.074d3eb0@ernst.home> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> Message-ID: <530A47BD.6040704@digiware.nl> On 23-2-2014 17:30, Gary Jennejohn wrote: > On Sun, 23 Feb 2014 10:18:31 -0500 (EST) > Daniel Eischen wrote: > >> On Sun, 23 Feb 2014, Willem Jan Withagen wrote: >> >>> On 16-2-2014 6:16, Perry Hutchison wrote: >>>> Jordan Hubbard wrote: >>>> >>>>> Even variant symlinks (/bin -> /${ARCH}/bin), which can expand >>>>> differently depending on the user context, have clearly >>>>> understandable semantics - you know that the symlink is going >>>>> to expand to exactly one file no matter what ARCH is set to. >>>> >>>> s/file/pathname/ >>>> >>>> Depending on what ARCH is set to, the expanision may or may not >>>> point to any actual file (or directory, or ...) >>> >>> Yes, please can we get these .... >>> >>> Apollo Domain systems had those, and they were great. >>> Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or >>> SYSV to get the other stuff. >>> >>> Would indeed work great for things like /bin or even >>> /usr/local/etc -> /${HOST}/usr/local/etc >> >> This topic comes up every couple of years. I recall >> Domain OS fondly - it was my first UNIX-like OS. I would >> really like variant symlinks, but I predict in another >> couple of years we'll be having the same conversation :-) >> > > Hear, hear! > > When I saw the first post I immediately thought "is it 1994 again?" > > Well, maybe the first discussion wasn't in 1994, but it was quite > some time ago. Should be around there when I took it up for the first time. Last dates on the code are from 1998, but I'm shure it did not work at that moment. It comes around in a regular cycle about every 7 years. :) --WjW From george+freebsd at m5p.com Sun Feb 23 19:36:56 2014 From: george+freebsd at m5p.com (George Mitchell) Date: Sun, 23 Feb 2014 14:36:48 -0500 Subject: Thoughts on Variant Symlinks; was Thoughts on Multi-Symlink Concept In-Reply-To: <530A47BD.6040704@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> Message-ID: <530A4DD0.2020904@m5p.com> Another veteran of Domain/OS here. Isn't it time to change the subject of this thread? Variant symlinks are quite distinct from what the OP was proposing (one "link" concatenates a plurality of files). -- George From adrian at freebsd.org Sun Feb 23 19:54:54 2014 From: adrian at freebsd.org (Adrian Chadd) Date: Sun, 23 Feb 2014 11:54:53 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <530A47BD.6040704@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> Message-ID: I like how noone mentioned how variant symlinks existed in the DEC UNIX on their alpha platforms. It made clustering stuff much more cute to do. -a From aduane at juniper.net Sun Feb 23 19:58:50 2014 From: aduane at juniper.net (Andrew Duane) Date: Sun, 23 Feb 2014 19:58:34 +0000 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> Message-ID: Don't get me started.... I lived that nightmare for years as a kernel dev at DEC/Compaq. IIRC, It was designed specifically for the clustering, and was a great big hunk of duct tape splatted on the side of the code. A seem to recall a little spackle around the edges too.... .................................... Andrew L. Duane AT&T Technical Lead JNCIA - JUNOS m???+1 603.770.7088 o +1 408.933.6944 (2-6944) skype: andrewlduane aduane at juniper.net -----Original Message----- From: owner-freebsd-hackers at freebsd.org [mailto:owner-freebsd-hackers at freebsd.org] On Behalf Of Adrian Chadd Sent: Sunday, February 23, 2014 2:55 PM To: Willem Jan Withagen Cc: freebsd-filesystems at freebsd.org; Perry Hutchison; freebsd-hackers at freebsd.org; Daniel Eischen; gljennjohn at gmail.com; Jordan Hubbard Subject: Re: Thoughts on Multi-Symlink Concept I like how noone mentioned how variant symlinks existed in the DEC UNIX on their alpha platforms. It made clustering stuff much more cute to do. -a _______________________________________________ freebsd-hackers at freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" From bakul at bitblocks.com Sun Feb 23 20:10:44 2014 From: bakul at bitblocks.com (Bakul Shah) Date: Sun, 23 Feb 2014 12:02:39 -0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> Message-ID: <4B5A7B51-74E2-4F8E-827D-251F0DBC9326@bitblocks.com> On Feb 22, 2014, at 7:13 PM, Jordan Hubbard wrote: > Indeed. I often tell interns who are looking for interesting project ideas to simply look back into our own past. Almost all the really interesting and cool research activities where operating systems are concerned seems to have happened between the years of 1970-1990. Sprite, Plan 9, Mach (hey, file space name servers anyone?), Domain OS, all kinds of neat ideas that sadly died or were forgotten in the name of consolidation, performance and expedience. Indeed, the performance of some of those concepts was actually rather woeful when 4MB of memory and 1MIP were all you one to work with. Maybe now that we have more hardware horsepower than we almost know what to do with, it?s time for some of those ideas to enjoy a renaissance? Sounds like a good EuroBSDCon or BSDCan talk. :-) Plan9 is alive and well! Well, at least used by a small but a diverse group of people! And with the advent of the plan9 RaspberryPi port I see new people playing with it! With Plan9 style per process name space, 9P protocol (to easily construct a filesystem API for anything), mount() syscall to connect to a fileserver speaking 9p, and bind() syscall to overlay/underlay a filetree on another, you can achieve the equivalent of variant symlinks and much more! You can easily implement a 'multi-symlink' fs as well! On plan9 all device drivers also speak 9p so you can even mount a remote network stack on your local machine (no need for NAT!). User programs such as rio (a window manager) and acme (an editor) also provide FS access to their facilities which makes it easy to write scripts to interact with them. Some examples: bind $ARCH/bin /bin # now files in $ARCH/bin appear in /bin mount /srv/dump /n/dump dump # make the dump fs available at /n/dump bind /n/dump/2013/11/12/arm/lib/libc.a /arm/lib/libc.a 5c -o foo foo.c # now foo is linked with libc.a of 12-Nov-2013 9fs sources # /n/sources points to sources/ on sources.cs.bell-labs.com bind /n/sources/plan9/sys/src /sys/src # overlay on local /sys/src There is already support for 9P in Linux, Qemu and few other places as it is pretty simple to add. UCB's many core research OS Akaros is using 9p and the network stack from plan9. If anyone is inspired to add 9p & friends support to FreeBSD, I encourage you to play with plan9 on the RaspberryPi as it is pretty easy to use and lots of fun. [Kernel compile takes a minute on the RasPi. The equivalent of `make buildworld buildkernel' about 4 minutes. Of course, the system is pretty minimal] From danny at cs.huji.ac.il Mon Feb 24 08:56:10 2014 From: danny at cs.huji.ac.il (Daniel Braniss) Date: Mon, 24 Feb 2014 10:39:10 +0200 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <4B5A7B51-74E2-4F8E-827D-251F0DBC9326@bitblocks.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <4B5A7B51-74E2-4F8E-827D-251F0DBC9326@bitblocks.com> Message-ID: and we also have automount/amd/am-utils/autofs ? here we have /usr/local via am-utils mounted to the correct arch/os/version. danny On Feb 23, 2014, at 10:02 PM, Bakul Shah wrote: > > On Feb 22, 2014, at 7:13 PM, Jordan Hubbard wrote: > >> Indeed. I often tell interns who are looking for interesting project ideas to simply look back into our own past. Almost all the really interesting and cool research activities where operating systems are concerned seems to have happened between the years of 1970-1990. Sprite, Plan 9, Mach (hey, file space name servers anyone?), Domain OS, all kinds of neat ideas that sadly died or were forgotten in the name of consolidation, performance and expedience. Indeed, the performance of some of those concepts was actually rather woeful when 4MB of memory and 1MIP were all you one to work with. Maybe now that we have more hardware horsepower than we almost know what to do with, it?s time for some of those ideas to enjoy a renaissance? Sounds like a good EuroBSDCon or BSDCan talk. :-) > > Plan9 is alive and well! Well, at least used by a small but > a diverse group of people! And with the advent of the plan9 > RaspberryPi port I see new people playing with it! > > With Plan9 style per process name space, 9P protocol (to > easily construct a filesystem API for anything), mount() > syscall to connect to a fileserver speaking 9p, and bind() > syscall to overlay/underlay a filetree on another, you can > achieve the equivalent of variant symlinks and much more! > You can easily implement a 'multi-symlink' fs as well! > > On plan9 all device drivers also speak 9p so you can even > mount a remote network stack on your local machine (no need > for NAT!). User programs such as rio (a window manager) and > acme (an editor) also provide FS access to their facilities > which makes it easy to write scripts to interact with them. > > Some examples: > > bind $ARCH/bin /bin # now files in $ARCH/bin appear in /bin > > mount /srv/dump /n/dump dump # make the dump fs available at /n/dump > bind /n/dump/2013/11/12/arm/lib/libc.a /arm/lib/libc.a > 5c -o foo foo.c # now foo is linked with libc.a of 12-Nov-2013 > > 9fs sources # /n/sources points to sources/ on sources.cs.bell-labs.com > bind /n/sources/plan9/sys/src /sys/src # overlay on local /sys/src > > There is already support for 9P in Linux, Qemu and few other > places as it is pretty simple to add. UCB's many core > research OS Akaros is using 9p and the network stack from > plan9. > > If anyone is inspired to add 9p & friends support to FreeBSD, > I encourage you to play with plan9 on the RaspberryPi as it is > pretty easy to use and lots of fun. [Kernel compile takes a > minute on the RasPi. The equivalent of `make buildworld > buildkernel' about 4 minutes. Of course, the system is pretty > minimal] > > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" From ilya at bakulin.de Mon Feb 24 14:26:51 2014 From: ilya at bakulin.de (Ilya Bakulin) Date: Mon, 24 Feb 2014 15:26:42 +0100 Subject: MMC/SDIO stack under CAM In-Reply-To: <5C2CF572-360D-4CA0-81C7-18A5C455AED5@bsdimp.com> References: <20140216111153.GA74858@olymp.kibab.com> <5C2CF572-360D-4CA0-81C7-18A5C455AED5@bsdimp.com> Message-ID: <20140224142642.GA32538@olymp.kibab.com> On Sun, Feb 16, 2014 at 11:05:35AM -0700, Warner Losh wrote: > that's cool! I'd thought of writing a mmcsim that I could use to develop the stack on x86, but time pressures of my job precluded that (though in hind sight, it would have saved a lot of time). I assume that by "mmcsim" you mean not MMC SIM for CAM, but MMC SIMulator :-) Yeah so essentially it should be able to return some "fake" data to the upper layer, maybe based on configuration (ie 1 GB SD, 16 GB SDHC, ...). > > For SDIO card, the situation will be different. Essentially, > > we should allow a device driver to send arbitrary messages to the card. > > So the device driver will attach directly to the scbus. > > > > Please let me know your thoughts about it. > > I really want SDIO make its way into the kernel, because there > > is an increasing number of ARM boards on the market that have integrated > > SDIO WLAN on them and we cannot support them fully. > > Generally, I like this idea... I'd be very interested in some of the specifics to make the direct attachment work with SDIO. That's the one area I either don't think I understand your proposal well enough, or I do and I'm concerned about it. :) So maybe write a bit more about the details of the SDIO cards and how they's interact with CAM in this scenario... > > The notion of moving MMC/SD into the CAM stack has been around since I started working on MMC. At the time, CAM was too SCSI centric to do it, but Alexander Motin's work has generally improved that situation, so it may be possible now to do it and shake out the few dark corners of CAM where this isn't the case. I also think it should help performance a lot since we're currently single threaded to the card (but that's also the nature of the bus), which introduces some round-trip latencies between too many threads that CAM may help us avoid. > > Warner After talking with Alexander about CAM, here is the updated proposal: * All card communication logic should be placed in MMC XPT layer, with the new CCB type that describes MMC request + some data needed by the system. So almost everything from sys/dev/mmc/mmc.c goes there. * da(4) is a driver for SCSI disks. It makes more sense to adapt mmcsd(4) to use CAM. So mmcsd(4) will create CCBs and pass them to CAM layer, getting async notifications via standard CAM callback mechanism. * It is possible to write drivers for SDIO devices exactly like adopted mmcsd(4). Every such driver would be then built as CAM peripheral driver, which means it will be able to send/receive CCBs. To do that, every driver has to call PERIPHDRIVER_DECLARE() macro, subscribing to CAM "new device found" events. The ongoing work for MMC CAM is here: https://github.com/kibab/freebsd/compare/master...mmccam Not so much stuff here yet. PS: In theory, ssince there is already an existing interface for passing CAM CCBs between userland and kernel, it WOULD be posssible to create device drivers entirely in userland... Like microkernel OSes do. But our network stack doesn'have such feature I guess... -- Ilya -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: From adrian.chadd at gmail.com Mon Feb 24 15:59:41 2014 From: adrian.chadd at gmail.com (Adrian Chadd) Date: Mon, 24 Feb 2014 07:59:40 -0800 Subject: MMC/SDIO stack under CAM In-Reply-To: <20140224142642.GA32538@olymp.kibab.com> References: <20140216111153.GA74858@olymp.kibab.com> <5C2CF572-360D-4CA0-81C7-18A5C455AED5@bsdimp.com> <20140224142642.GA32538@olymp.kibab.com> Message-ID: hi, Let me just reiterate some .. well, experience doing this stuff at QCA. You really, absolutely don't want too much overhead in the MMC/SDIO path between whatever is issuing things and the network driver. There was significant performance work done at QCA on a local MMC/SDIO driver and bus to get extremely low latency and CPU utilisation when pushing around small transactions. The current CAM locking model is not geared towards getting to high transaction rates. You may think this is a very architecturally pretty solution and it indeed may be. But if it doesn't perform as well as the existing local hacks that vendors have done, no company deploying this hardware is going to want to use it. They'll end up realising there's this massive CAM storage layer in between and either have to sit down to rip it up and replace it with something lightweight, or they'll say "screw it" and go back to the vendor supplied hacked up Linux solution. So I highly recommend you profile things - and profile things with lots of small transactions. If the CAM overhead is more than a tiny, tiny fraction of CPU at 25,000 pps, your solution won't scale. :-) -a On 24 February 2014 06:26, Ilya Bakulin wrote: > On Sun, Feb 16, 2014 at 11:05:35AM -0700, Warner Losh wrote: >> that's cool! I'd thought of writing a mmcsim that I could use to develop the stack on x86, but time pressures of my job precluded that (though in hind sight, it would have saved a lot of time). > > I assume that by "mmcsim" you mean not MMC SIM for CAM, but MMC SIMulator :-) > Yeah so essentially it should be able to return some "fake" data > to the upper layer, maybe based on configuration (ie 1 GB SD, 16 GB SDHC, ...). > >> > For SDIO card, the situation will be different. Essentially, >> > we should allow a device driver to send arbitrary messages to the card. >> > So the device driver will attach directly to the scbus. >> > >> > Please let me know your thoughts about it. >> > I really want SDIO make its way into the kernel, because there >> > is an increasing number of ARM boards on the market that have integrated >> > SDIO WLAN on them and we cannot support them fully. >> >> Generally, I like this idea... I'd be very interested in some of the specifics to make the direct attachment work with SDIO. That's the one area I either don't think I understand your proposal well enough, or I do and I'm concerned about it. :) So maybe write a bit more about the details of the SDIO cards and how they's interact with CAM in this scenario... >> >> The notion of moving MMC/SD into the CAM stack has been around since I started working on MMC. At the time, CAM was too SCSI centric to do it, but Alexander Motin's work has generally improved that situation, so it may be possible now to do it and shake out the few dark corners of CAM where this isn't the case. I also think it should help performance a lot since we're currently single threaded to the card (but that's also the nature of the bus), which introduces some round-trip latencies between too many threads that CAM may help us avoid. >> >> Warner > > After talking with Alexander about CAM, here is the updated proposal: > > * All card communication logic should be placed in MMC XPT layer, with the new CCB type > that describes MMC request + some data needed by the system. > So almost everything from sys/dev/mmc/mmc.c goes there. > * da(4) is a driver for SCSI disks. It makes more sense to adapt mmcsd(4) to use CAM. > So mmcsd(4) will create CCBs and pass them to CAM layer, getting async notifications > via standard CAM callback mechanism. > * It is possible to write drivers for SDIO devices exactly like adopted mmcsd(4). > Every such driver would be then built as CAM peripheral driver, which means it will be able > to send/receive CCBs. > To do that, every driver has to call PERIPHDRIVER_DECLARE() macro, subscribing to CAM "new device found" events. > > The ongoing work for MMC CAM is here: > https://github.com/kibab/freebsd/compare/master...mmccam > Not so much stuff here yet. > > PS: In theory, ssince there is already an existing interface for passing CAM CCBs between userland and kernel, > it WOULD be posssible to create device drivers entirely in userland... Like microkernel OSes do. > But our network stack doesn'have such feature I guess... > > -- > Ilya From rwatson at FreeBSD.org Mon Feb 24 18:31:09 2014 From: rwatson at FreeBSD.org (Robert Watson) Date: Mon, 24 Feb 2014 18:31:08 +0000 (GMT) Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> Message-ID: On Sat, 22 Feb 2014, Jordan Hubbard wrote: >> Yes, please can we get these .... >> >> Apollo Domain systems had those, and they were great. Set SYSTYPE to BSD4 >> and get the BSD tree and all that came with it, or SYSV to get the other >> stuff. > > Yep, I loved these things on Domain/OS! We system admin types used them to > do all kinds of clever (and useful) things. > > Looks like FreeBSD has actually *had* an implementation for 6 years now. I > don?t necessarily agree with the architectural decision to create a > different namespace and command (varsym) to manipulate it - it was really > nice just having it be a part of the standard environ(7) - but hey, any > implementation is better than no implementation. Whatever happened to > https://wiki.freebsd.org/200808DevSummit?action=AttachFile&do=get&target=variant-symlinks-for-freebsd.pdf > ? Some care is required here: at least one of the past implementations floating around had the neat property that user-defined symlink expansions occurred before system-defined ones, even for setuid binaries. This allowed trivial replacement of libraries out from under a binary, making rooting boxes easy. I'm actually a fan of variant symlinks as well, having used them in AFS -- I'd just prefer we aim for a model that minimises inconvenient rooting of boxes. (I'm not passing judgement on this particular patch, mind you.) I believe Brooks Davis did the last serious pass at variant symlinks and might opine further on the topic. Robert From aduane at juniper.net Mon Feb 24 18:37:54 2014 From: aduane at juniper.net (Andrew Duane) Date: Mon, 24 Feb 2014 18:37:39 +0000 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> Message-ID: -----Original Message----- From: owner-freebsd-hackers at freebsd.org [mailto:owner-freebsd-hackers at freebsd.org] On Behalf Of Robert Watson Sent: Monday, February 24, 2014 1:31 PM To: Jordan Hubbard Cc: freebsd-filesystems at freebsd.org; freebsd-hackers at freebsd.org; Willem Jan Withagen; Perry Hutchison Subject: Re: Thoughts on Multi-Symlink Concept On Sat, 22 Feb 2014, Jordan Hubbard wrote: >>> Apollo Domain systems had those, and they were great. Set SYSTYPE to BSD4 >>> and get the BSD tree and all that came with it, or SYSV to get the other >.> stuff. >> >> Yep, I loved these things on Domain/OS! We system admin types used them to >> do all kinds of clever (and useful) things. >> >> Looks like FreeBSD has actually *had* an implementation for 6 years now. I >> don't necessarily agree with the architectural decision to create a >> different namespace and command (varsym) to manipulate it - it was really >> nice just having it be a part of the standard environ(7) - but hey, any >> implementation is better than no implementation. Whatever happened to >> https://wiki.freebsd.org/200808DevSummit?action=AttachFile&do=get&target=variant-symlinks-for-freebsd.pdf? > >Some care is required here: at least one of the past implementations floating >around had the neat property that user-defined symlink expansions occurred >before system-defined ones, even for setuid binaries. This allowed trivial >replacement of libraries out from under a binary, making rooting boxes easy. >I'm actually a fan of variant symlinks as well, having used them in AFS -- I'd >just prefer we aim for a model that minimises inconvenient rooting of boxes. >(I'm not passing judgement on this particular patch, mind you.) I believe >Brooks Davis did the last serious pass at variant symlinks and might opine >further on the topic. > >Robert I'd also be careful of violating the Principle of Least Astonishment with any Implementation. Multi or Variable symlnks that suddenly change meaning Can really confound people. /Andrew From brooks at freebsd.org Mon Feb 24 19:46:16 2014 From: brooks at freebsd.org (Brooks Davis) Date: Mon, 24 Feb 2014 13:46:08 -0600 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> Message-ID: <20140224194608.GD18404@lor.one-eyed-alien.net> On Mon, Feb 24, 2014 at 06:31:08PM +0000, Robert Watson wrote: > On Sat, 22 Feb 2014, Jordan Hubbard wrote: > > >> Yes, please can we get these .... > >> > >> Apollo Domain systems had those, and they were great. Set SYSTYPE to BSD4 > >> and get the BSD tree and all that came with it, or SYSV to get the other > >> stuff. > > > > Yep, I loved these things on Domain/OS! We system admin types used them to > > do all kinds of clever (and useful) things. > > > > Looks like FreeBSD has actually *had* an implementation for 6 years now. I > > don?t necessarily agree with the architectural decision to create a > > different namespace and command (varsym) to manipulate it - it was really > > nice just having it be a part of the standard environ(7) - but hey, any > > implementation is better than no implementation. Whatever happened to > > https://wiki.freebsd.org/200808DevSummit?action=AttachFile&do=get&target=variant-symlinks-for-freebsd.pdf > > ? > > Some care is required here: at least one of the past implementations floating > around had the neat property that user-defined symlink expansions occurred > before system-defined ones, even for setuid binaries. This allowed trivial > replacement of libraries out from under a binary, making rooting boxes easy. > I'm actually a fan of variant symlinks as well, having used them in AFS -- I'd > just prefer we aim for a model that minimises inconvenient rooting of boxes. > (I'm not passing judgement on this particular patch, mind you.) I believe > Brooks Davis did the last serious pass at variant symlinks and might opine > further on the topic. The version I wrote worked find last time I ported it forward (20 months or so ago). There's a copy in svn that's the latest. The thing that's held me back from committing it is lack to time to do some solid macro and micro benchmarks to determine what the performance impact is when it's compiled in, but disabled. It should probably just be updated, reviewed by a VFS person and committed. I believe I've successfully avoided the pitfalls Robert describes. -- Brooks -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 326 bytes Desc: not available URL: From wkoszek at freebsd.org Mon Feb 24 23:21:28 2014 From: wkoszek at freebsd.org (Wojciech A. Koszek) Date: Mon, 24 Feb 2014 23:20:22 +0000 Subject: FreeBSD in GSoC 2014! (mentors wanted) Message-ID: <20140224232022.GM47836@FreeBSD.org> (cross-posted message; keep discussion on hackers@ only) Hello, So we're in GSOC 2014! Our logo is featured on the main website: http://www.google-melange.com/gsoc/homepage/google/gsoc2014 Everybody can submit ideas through a web form: http://tinyurl.com/FreeBSD-GSOC2014 To help, please add/review/revisit ideas from the FreeBSD Wiki and provide mentorship! https://wiki.freebsd.org/SummerOfCode2014 There are ideas without mentors and ideas with only 1 mentor, as well as tasks which haven't been reviewed.. Help would be appreciated, Thanks, -- Wojciech A. Koszek wkoszek at FreeBSD.czest.pl http://FreeBSD.czest.pl/~wkoszek/ From george+freebsd at m5p.com Tue Feb 25 00:18:37 2014 From: george+freebsd at m5p.com (George Mitchell) Date: Mon, 24 Feb 2014 19:17:54 -0500 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> Message-ID: <530BE132.1030507@m5p.com> On 02/22/14 21:45, Jordan Hubbard wrote: > > On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen wrote: > >> Yes, please can we get these .... >> >> Apollo Domain systems had those, and they were great. >> Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or >> SYSV to get the other stuff. > [...] Since we're going down DomainOS Memory Lane, does anyone else miss transcript pads? -- George From deischen at freebsd.org Tue Feb 25 01:20:50 2014 From: deischen at freebsd.org (Daniel Eischen) Date: Mon, 24 Feb 2014 20:20:41 -0500 (EST) Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <530BE132.1030507@m5p.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> <530BE132.1030507@m5p.com> Message-ID: On Mon, 24 Feb 2014, George Mitchell wrote: > On 02/22/14 21:45, Jordan Hubbard wrote: >> >> On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen wrote: >> >>> Yes, please can we get these .... >>> >>> Apollo Domain systems had those, and they were great. >>> Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or >>> SYSV to get the other stuff. >> [...] > > Since we're going down DomainOS Memory Lane, does anyone else miss > transcript pads? -- George I missed the transcript pad too. I also missed the DM editor and the ability to write DM scripts. And I miss the edit pad - the first time I was able to do a rectangular cut/paste/move. And I loved the ability to place the cursor beyond the current end of a line and start typing (without clicking the mouse!). I hate having to hold down the space bar, adding blanks, to get out to the column I want (suppose your coding conventions disallow tabs). -- DE From julian at freebsd.org Tue Feb 25 02:55:37 2014 From: julian at freebsd.org (Julian Elischer) Date: Tue, 25 Feb 2014 10:55:24 +0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <530A47BD.6040704@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> Message-ID: <530C061C.9020803@freebsd.org> On 2/24/14, 3:10 AM, Willem Jan Withagen wrote: > On 23-2-2014 17:30, Gary Jennejohn wrote: >> On Sun, 23 Feb 2014 10:18:31 -0500 (EST) >> Daniel Eischen wrote: >> >>> On Sun, 23 Feb 2014, Willem Jan Withagen wrote: >>> >>>> On 16-2-2014 6:16, Perry Hutchison wrote: >>>>> Jordan Hubbard wrote: >>>>> >>>>>> Even variant symlinks (/bin -> /${ARCH}/bin), which can expand >>>>>> differently depending on the user context, have clearly >>>>>> understandable semantics - you know that the symlink is going >>>>>> to expand to exactly one file no matter what ARCH is set to. >>>>> s/file/pathname/ >>>>> >>>>> Depending on what ARCH is set to, the expanision may or may not >>>>> point to any actual file (or directory, or ...) >>>> Yes, please can we get these .... >>>> >>>> Apollo Domain systems had those, and they were great. >>>> Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or >>>> SYSV to get the other stuff. >>>> >>>> Would indeed work great for things like /bin or even >>>> /usr/local/etc -> /${HOST}/usr/local/etc >>> This topic comes up every couple of years. I recall >>> Domain OS fondly - it was my first UNIX-like OS. I would >>> really like variant symlinks, but I predict in another >>> couple of years we'll be having the same conversation :-) >>> >> Hear, hear! >> >> When I saw the first post I immediately thought "is it 1994 again?" >> >> Well, maybe the first discussion wasn't in 1994, but it was quite >> some time ago. > Should be around there when I took it up for the first time. > Last dates on the code are from 1998, but I'm shure it did not work at > that moment. > > It comes around in a regular cycle about every 7 years. :) oh it was way before 1994.. believe me. It even came before fairings. > > --WjW > > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" > From julian at freebsd.org Tue Feb 25 03:01:13 2014 From: julian at freebsd.org (Julian Elischer) Date: Tue, 25 Feb 2014 11:01:02 +0800 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> <530BE132.1030507@m5p.com> Message-ID: <530C076E.2030802@freebsd.org> On 2/25/14, 9:20 AM, Daniel Eischen wrote: > On Mon, 24 Feb 2014, George Mitchell wrote: > >> On 02/22/14 21:45, Jordan Hubbard wrote: >>> >>> On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen >>> wrote: >>> >>>> Yes, please can we get these .... >>>> >>>> Apollo Domain systems had those, and they were great. >>>> Set SYSTYPE to BSD4 and get the BSD tree and all that came with >>>> it, or >>>> SYSV to get the other stuff. >>> [...] >> >> Since we're going down DomainOS Memory Lane, does anyone else miss >> transcript pads? -- George > > I missed the transcript pad too. > > I also missed the DM editor and the ability to write DM scripts. > > And I miss the edit pad - the first time I was able to do > a rectangular cut/paste/move. And I loved the ability to place > the cursor beyond the current end of a line and start typing > (without clicking the mouse!). I hate having to hold down > the space bar, adding blanks, to get out to the column I want > (suppose your coding conventions disallow tabs). > I miss teco From jim at netgate.com Tue Feb 25 03:08:12 2014 From: jim at netgate.com (Jim Thompson) Date: Mon, 24 Feb 2014 21:07:53 -0600 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: <530C076E.2030802@freebsd.org> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> <530BE132.1030507@m5p.com> <530C076E.2030802@freebsd.org> Message-ID: <67DDCEAC-9046-474B-BAAE-E4CF54CCFC64@netgate.com> On Feb 24, 2014, at 9:01 PM, Julian Elischer wrote: > I miss teco You must mean ?Expensive Typewriter? From jhellenthal at dataix.net Tue Feb 25 05:34:06 2014 From: jhellenthal at dataix.net (Jason Hellenthal) Date: Mon, 24 Feb 2014 23:56:21 -0500 Subject: Thoughts on Multi-Symlink Concept In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <4B5A7B51-74E2-4F8E-827D-251F0DBC9326@bitblocks.com> Message-ID: <1447F7F5-8030-4340-B4E3-7A3EAF8055D3@dataix.net> That's a bit more complicated than the average Yogi bear would look for though. -- Jason Hellenthal Voice: 95.30.17.6/616 JJH48-ARIN > On Feb 24, 2014, at 3:39, Daniel Braniss wrote: > > and we also have automount/amd/am-utils/autofs ? > > here we have /usr/local via am-utils mounted to the correct arch/os/version. > > danny > > >> On Feb 23, 2014, at 10:02 PM, Bakul Shah wrote: >> >> >>> On Feb 22, 2014, at 7:13 PM, Jordan Hubbard wrote: >>> >>> Indeed. I often tell interns who are looking for interesting project ideas to simply look back into our own past. Almost all the really interesting and cool research activities where operating systems are concerned seems to have happened between the years of 1970-1990. Sprite, Plan 9, Mach (hey, file space name servers anyone?), Domain OS, all kinds of neat ideas that sadly died or were forgotten in the name of consolidation, performance and expedience. Indeed, the performance of some of those concepts was actually rather woeful when 4MB of memory and 1MIP were all you one to work with. Maybe now that we have more hardware horsepower than we almost know what to do with, it?s time for some of those ideas to enjoy a renaissance? Sounds like a good EuroBSDCon or BSDCan talk. :-) >> >> Plan9 is alive and well! Well, at least used by a small but >> a diverse group of people! And with the advent of the plan9 >> RaspberryPi port I see new people playing with it! >> >> With Plan9 style per process name space, 9P protocol (to >> easily construct a filesystem API for anything), mount() >> syscall to connect to a fileserver speaking 9p, and bind() >> syscall to overlay/underlay a filetree on another, you can >> achieve the equivalent of variant symlinks and much more! >> You can easily implement a 'multi-symlink' fs as well! >> >> On plan9 all device drivers also speak 9p so you can even >> mount a remote network stack on your local machine (no need >> for NAT!). User programs such as rio (a window manager) and >> acme (an editor) also provide FS access to their facilities >> which makes it easy to write scripts to interact with them. >> >> Some examples: >> >> bind $ARCH/bin /bin # now files in $ARCH/bin appear in /bin >> >> mount /srv/dump /n/dump dump # make the dump fs available at /n/dump >> bind /n/dump/2013/11/12/arm/lib/libc.a /arm/lib/libc.a >> 5c -o foo foo.c # now foo is linked with libc.a of 12-Nov-2013 >> >> 9fs sources # /n/sources points to sources/ on sources.cs.bell-labs.com >> bind /n/sources/plan9/sys/src /sys/src # overlay on local /sys/src >> >> There is already support for 9P in Linux, Qemu and few other >> places as it is pretty simple to add. UCB's many core >> research OS Akaros is using 9p and the network stack from >> plan9. >> >> If anyone is inspired to add 9p & friends support to FreeBSD, >> I encourage you to play with plan9 on the RaspberryPi as it is >> pretty easy to use and lots of fun. [Kernel compile takes a >> minute on the RasPi. The equivalent of `make buildworld >> buildkernel' about 4 minutes. Of course, the system is pretty >> minimal] >> >> >> _______________________________________________ >> freebsd-hackers at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6118 bytes Desc: not available URL: From wjw at digiware.nl Tue Feb 25 08:20:15 2014 From: wjw at digiware.nl (Willem Jan Withagen) Date: Tue, 25 Feb 2014 09:20:11 +0100 Subject: Variant Links (Was: Thoughts on Multi-Symlink Concept) In-Reply-To: <530C061C.9020803@freebsd.org> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> <530C061C.9020803@freebsd.org> Message-ID: <530C523B.1000802@digiware.nl> On 2014-02-25 3:55, Julian Elischer wrote: > On 2/24/14, 3:10 AM, Willem Jan Withagen wrote: >>> Well, maybe the first discussion wasn't in 1994, but it was quite >>> some time ago. >> Should be around there when I took it up for the first time. >> Last dates on the code are from 1998, but I'm shure it did not work at >> that moment. >> >> It comes around in a regular cycle about every 7 years. :) > > oh it was way before 1994.. > believe me. It even came before fairings. Perhaps I should have said: My first discussion in FreeBSD lists. Note that I looked at pictures of an office of a citi.umich DomainOS-friend of mine, and there was still a Apollo Domain system. A bit yellowish, and with the cover removed... ot shure if his actually still runs it. --WjW From ghostman.sd at gmail.com Tue Feb 25 08:24:39 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Tue, 25 Feb 2014 12:24:17 +0400 Subject: GSoC proposal: Quirinus C library (qc) Message-ID: Hello everyone! My name is Dmitry, I'm 22 years old student from Lomonosov Moscow State University of Russia. This message is addressed mainly to C connoiseurs, yet I think other people may find it interesting. It's a GSoC proposal. It's a pity that for language like C we generally don't have something universal like Boost, so we have to implement some common functions from scratch or introduce new dependencies. We have Glib, which is used mainly by Gnome developers (though it is a standalone library) and LGPL-ed, which is not as liberal as Boost's license. We also have APR, which seems to be a bit more comprehensive and convenient, yet it is not so well-known as Glib. I'm in process of implementing a something like Boost for ANSI C (though I don't pretend this library to share Boost's comprehensiveness). Several ideas I find useful are: 1. Strict and universal interface. Each function begins with qc_ prefix, followed by type if function is type-related. Most of types (except enum-typedef'ed ones) have three common functions (replace _type_ with the necessary type name, e.g. _bytes_): a. Constructor: void * qc_type_construct(void). Allocate object instance and initialize it to default value. b. Destructor: void qc_type_destruct(void * pointer). Deallocate memory allocated for object and its members. c. Replicator: void * qc_type_replicate(void const * pointer). Replicate object, copying its data to new allocated object, i.e. clone. Most of types also have _import_ and _export_ functions, which allow to fill allocated object with the desired data (e.g. fill bytes array from null-terminated char string) or export data to another type. Almost all enum-typedef'ed types also have _import_ and _export_ functions to retrieve a key value corresponding to string. 2. Common and universal error type, qc_error. Such errors can be generated from errno values as well as from Windows API errors. Almost all functions from qc library except for the three common functions (constructor, destructor, replicator) return qc_error code and set specific qc_errno variable to this code. It is now possible to write such constructions: if (qc_bytes_import_str(bytes_instance, "Hello, world!")) qc_errormsg(qc_errno, "error check"); Global variable qc_errno is implemented in terms of multithreading applications, it is unique for every running thread. 3. Clear distinction between byte and Unicode strings (qc_byte and qc_ucs types for characters, where qc_byte has exactly 8-bit width and qc_ucs has exactly 32-bit width). Charsets are just enumeration, e.g. QC_ENCODING_UTF8, QC_ENCODING_ISO_8859_4, etc., yet it is possible to lookup the desired encoding using qc_encoding_import. If encoding is known under several names, they are handled alltogether, i.e. QC_ENCODING_ASCII will be returned for any of "ASCII", "iso-ir-6", "ANSI_X3.4-1968", "ISO646-US", etc. Register doesn't matter. Two new types, qc_bytes and qc_unicode are provided, in order to store binary and Unicode data. It is possible to store null characters inside such objects. It is similar to C++ string and C++ vector classes. 4. Universal file system path type. It is possible to achieve cross-platformness using such path types, i.e. it is possible to make directories, links, symlinks, remove files, directories, etc. regardless of platform path API. 5. i18n support must be embedded with qc library. Locales, timezones, day and months names, etc. must be provided too, in several forms if necessary. i18n functions work with qc_unicode type, so charset conversion problems won't exist. 6. Multithreading support if platform permits it. On POSIX we use pthreads, on Windows we use its API for multithreading. I'm also thinking about green threads implementation. Thread local storage is already implemented, yet there is still a lot work to be done with threads, mutexes, etc. 7. Multiprecision arithmetics. I'm using GMP to achieve it, yet I'm planning to switch to something BSD-like or to implement it from scratch (that probably would be better, since library doesn't introduce any dependencies except GMP). 8. Ternary logic almost everywhere. Being fan of Russian Setun computer, I've started this library as implementation of ternary logic operations. When I've realized that there is still much to be done in other fields, I've already concluded that ternary logic is suitable for a wide set of other operations, e.g. comparison, determining type of strings, endianness, etc. I think that I'll leave type qc_trit, since I've found it extremely useful. 9. A lot of other things to be done, such as unified I/O streams which provide compression/transformation filters, protocols, math library, etc. Why I'm writing such letter to FreeBSD? I'm pretty sure that FreeBSD highly needs such library, since we try to use BSD-like license and yet to implement things that exist in GNU world. The most common example is GNU readline library, yet I think there is a lot of libraries that we reimplemented just to let some projects suit BSD philosophy. We can make a good library, which can be used a lot in different projects, yet it will field everyone like Boost. The nature of C language itself helps us to keep this library much more tiny than Boost library; I also think that we can limit our support with POSIX-(almost-)compatible platforms and Windows. I've already done a lot of work, but it is hard to make decisions without advice or discussion, so I've thought about to enter GSoC this year like I did two years ago when I've reimplemented gnulib-tool for FSF. This project is much more interesting for me, since I want to upgrade my C skills, get known BSD world better and provide a good cross-platform library which may be useful to almost everyone. Since I've already finished my volunteer's job in Sochi (I've worked as technologist volunteer), I'm really want to make something new for other people to be useful. If you are interested in my project, here is repository at GitHub: https://github.com/ghostmansd/quirinus. I guess I'll later rename it to https://github.com/ghostmansd/qc, yet it doesn't matters. If you're interested in my previous GSoC work, you can find it here: https://github.com/ghostmansd/gnulib-python. If you want to get to know me better from any side (my philological studies in Lomonosov Moscow University, GSoC participation, Sochi volunteer job, etc.), you can write me a email; I'd be very glad. Thanks for reading such a long letter! -- With best regards, Dmitry Selyutin From jordan.hubbard at gmail.com Tue Feb 25 18:28:59 2014 From: jordan.hubbard at gmail.com (Jordan Hubbard) Date: Tue, 25 Feb 2014 10:28:54 -0800 Subject: Variant Links (Was: Thoughts on Multi-Symlink Concept) In-Reply-To: <530C523B.1000802@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> <530C061C.9020803@freebsd.org> <530C523B.1000802@digiware.nl> Message-ID: <77202023-992F-4333-9CD8-B6A73A8BE8F7@gmail.com> On Feb 25, 2014, at 12:20 AM, Willem Jan Withagen wrote: > Perhaps I should have said: My first discussion in FreeBSD lists. Welcome to the endless digression! I think this is actually a pretty good introduction since many of our conversations go like this (we haven't yet had the "Will you guys just SHUT UP about DomainOS already? I wasn't even born when people were using that thing - this is BORING!" message - I'm sure it's coming :-) ) - Jordan From jim at netgate.com Tue Feb 25 20:18:29 2014 From: jim at netgate.com (Jim Thompson) Date: Tue, 25 Feb 2014 14:18:22 -0600 Subject: Variant Links (Was: Thoughts on Multi-Symlink Concept) In-Reply-To: <77202023-992F-4333-9CD8-B6A73A8BE8F7@gmail.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> <530C061C.9020803@freebsd.org> <530C523B.1000802@digiware.nl> <77202023-992F-4333-9CD8-B6A73A8BE8F7@gmail.com> Message-ID: On Feb 25, 2014, at 12:28 PM, Jordan Hubbard wrote: > "Will you guys just SHUT UP about DomainOS already? I wasn't even born when people were using that thing - this is BORING!" You must have missed the TECO branch of the discussion. Jim From jim at netgate.com Tue Feb 25 20:19:07 2014 From: jim at netgate.com (Jim Thompson) Date: Tue, 25 Feb 2014 14:18:22 -0600 Subject: Variant Links (Was: Thoughts on Multi-Symlink Concept) In-Reply-To: <77202023-992F-4333-9CD8-B6A73A8BE8F7@gmail.com> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <20140223173042.074d3eb0@ernst.home> <530A47BD.6040704@digiware.nl> <530C061C.9020803@freebsd.org> <530C523B.1000802@digiware.nl> <77202023-992F-4333-9CD8-B6A73A8BE8F7@gmail.com> Message-ID: <265259F0-A805-4ABE-B4E5-57F61793EE07@netgate.com> On Feb 25, 2014, at 12:28 PM, Jordan Hubbard wrote: > "Will you guys just SHUT UP about DomainOS already? I wasn't even born when people were using that thing - this is BORING!" You must have missed the TECO branch of the discussion. Jim From lev at FreeBSD.org Tue Feb 25 21:31:20 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Wed, 26 Feb 2014 01:31:07 +0400 Subject: What is difference between loading module with loader and loading module wtih kldload? Message-ID: <9890815.20140226013107@serebryakov.spb.ru> Hello, Hackers. I've upgraded my fileserver to 10-STABLE and got very strange (but very-very painful) problem, which I could not reproduce on virtual machine (VirtualBox). I'm using geom_raid5 module (and I'm its maintainer, yes) and module, built for 10-STABLE (after world & kernel build & install & reboot), and loaded via /boot/loader.conf is reason to almost instant crash after boot. Sometimes system mounts filesystems before crash and sometimes not. Most of time it is "page write: page not present", in different places. PS/2 keyboard is always blocked after that, I could not drop to debugger. No memory dump performed. Several times it turend off video output (!) right after crash. But if I boot without this module, drop to single-user mode, load module with kldload and continue booting with "exit" everything work smoothly for hours! I understand, that it it some incompatibility between module new kernel, but I could not reproduce it on VirtualBox instance, and I'm puzzled, that this crash does not occur if module loaded by kldload! Maybe, here is some hint in this? -- // Black Lion AKA Lev Serebryakov From bsd-lists at bsdforge.com Tue Feb 25 23:21:09 2014 From: bsd-lists at bsdforge.com (Chris H) Date: Tue, 25 Feb 2014 15:21:16 -0800 (PST) Subject: What is difference between loading module with loader and loading module wtih kldload? In-Reply-To: <9890815.20140226013107@serebryakov.spb.ru> References: <9890815.20140226013107@serebryakov.spb.ru> Message-ID: <9cb58e6ded37c79769da936cf17087da.authenticated@ultimatedns.net> > Hello, Hackers. > > I've upgraded my fileserver to 10-STABLE and got very strange (but > very-very painful) problem, which I could not reproduce on virtual machine > (VirtualBox). > > I'm using geom_raid5 module (and I'm its maintainer, yes) and module, built > for 10-STABLE (after world & kernel build & install & reboot), and loaded > via /boot/loader.conf is reason to almost instant crash after boot. > Sometimes system mounts filesystems before crash and sometimes not. Most of > time it is "page write: page not present", in different places. PS/2 > keyboard is always blocked after that, I could not drop to debugger. No > memory dump performed. Several times it turend off video output (!) right > after crash. > > But if I boot without this module, drop to single-user mode, load module > with kldload and continue booting with "exit" everything work smoothly for > hours! > > I understand, that it it some incompatibility between module new kernel, > but I could not reproduce it on VirtualBox instance, and I'm puzzled, that > this crash does not occur if module loaded by kldload! Maybe, here is some > hint in this? Warning, I'm no expert on 10.x. I see too many issues by ppl on the lists. So it seems too shakey for me. But, I have a couple of ideas, hoping they might help. I had a similar problem when using a very large drive, or mirror set (geom), a few years ago. FreeBSD didn't see/allocate all the memory available on the system. So it barfed, because it didn't have enough to allocate to handle the drive size. I'm sorry, but I have forgotten how to TELL FreeBSD how much memory it can use during boot tho. But if you know, then this might help you too. Hope this helps. --Chris > > -- > // Black Lion AKA Lev Serebryakov > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" > From bsd-lists at bsdforge.com Tue Feb 25 23:31:40 2014 From: bsd-lists at bsdforge.com (Chris H) Date: Tue, 25 Feb 2014 15:31:56 -0800 (PST) Subject: How to prevent use of clang && llvm? Message-ID: <47ae4e8ed445c7e6fc46796d46b9a928.authenticated@ultimatedns.net> Greetings, I'm in the process of performing an upgrade, and ran a preliminary check to see what would be performed. During the process I encountered a dialog(1) for both clang, and llvm3. It was for textproc/clucene. Which wants devel/boost-libs, which depends on devel/boost-jam. I didn't require clang, or llvm3 with the initial install of textproc/clucene (that I can see/recall). But am presented with the dialog this time. I have WITHOUT_CLANG=true in /etc/src.conf. Do I need to also add it to /etc/make.conf? Anything else? Thank you for all your time, and consideration. --Chris From brooks at freebsd.org Wed Feb 26 00:04:37 2014 From: brooks at freebsd.org (Brooks Davis) Date: Tue, 25 Feb 2014 18:04:21 -0600 Subject: How to prevent use of clang && llvm? In-Reply-To: <47ae4e8ed445c7e6fc46796d46b9a928.authenticated@ultimatedns.net> References: <47ae4e8ed445c7e6fc46796d46b9a928.authenticated@ultimatedns.net> Message-ID: <20140226000421.GA80747@lor.one-eyed-alien.net> On Tue, Feb 25, 2014 at 03:31:56PM -0800, Chris H wrote: > Greetings, > I'm in the process of performing an upgrade, and ran a preliminary check > to see what would be performed. During the process I encountered a > dialog(1) for both clang, and llvm3. It was for textproc/clucene. Which > wants devel/boost-libs, which depends on devel/boost-jam. I didn't require > clang, or llvm3 with the initial install of textproc/clucene (that I can see/recall). > But am presented with the dialog this time. I have WITHOUT_CLANG=true > in /etc/src.conf. Do I need to also add it to /etc/make.conf? Anything else? It looks like this is because boots requires a C++11 compiler and the default C++11 compiler is clang. From reading the source I deduce that FAVORITE_COMPILER=gcc in your make.conf may achieve the effect you require. -- Brooks -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 326 bytes Desc: not available URL: From jmg at funkthat.com Wed Feb 26 00:12:41 2014 From: jmg at funkthat.com (John-Mark Gurney) Date: Tue, 25 Feb 2014 16:12:38 -0800 Subject: What is difference between loading module with loader and loading module wtih kldload? In-Reply-To: <9890815.20140226013107@serebryakov.spb.ru> References: <9890815.20140226013107@serebryakov.spb.ru> Message-ID: <20140226001238.GS92037@funkthat.com> Lev Serebryakov wrote this message on Wed, Feb 26, 2014 at 01:31 +0400: > I've upgraded my fileserver to 10-STABLE and got very strange (but > very-very painful) problem, which I could not reproduce on virtual machine > (VirtualBox). > > I'm using geom_raid5 module (and I'm its maintainer, yes) and module, built > for 10-STABLE (after world & kernel build & install & reboot), and loaded > via /boot/loader.conf is reason to almost instant crash after boot. > Sometimes system mounts filesystems before crash and sometimes not. Most of > time it is "page write: page not present", in different places. PS/2 > keyboard is always blocked after that, I could not drop to debugger. No > memory dump performed. Several times it turend off video output (!) right > after crash. Can you give us an exact error message? I am not finding the string: "page write: page not present" anywhere in the tree, on close thing is in trap, where it could be user/supervisor write/read instruction/data page not present, where words seperated by slashes could be one or the other... This sounds like it could be a buffer overflow... Could you try turning on INVARIANTS and other related debugging on 10-STABLE since these were turned off for the RELEASE? Also uname -a would be helpful to know which arch you are on... > But if I boot without this module, drop to single-user mode, load module > with kldload and continue booting with "exit" everything work smoothly for > hours! > > I understand, that it it some incompatibility between module new kernel, > but I could not reproduce it on VirtualBox instance, and I'm puzzled, that > this crash does not occur if module loaded by kldload! Maybe, here is some > hint in this? There are a few differences between a boot time loaded module and a runtime loaded module... The linker runs earlier at boot time to link up the module before things start, the module may be partly run w/ cold set (which informs us the interrupts and other things may not be fully working).. Sysinits are run in a slightly different order (i.e. kernel SYSINITs that appear after your modules will be run first)... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From bsd-lists at bsdforge.com Wed Feb 26 00:19:46 2014 From: bsd-lists at bsdforge.com (Chris H) Date: Tue, 25 Feb 2014 16:20:02 -0800 (PST) Subject: How to prevent use of clang && llvm? In-Reply-To: <20140226000421.GA80747@lor.one-eyed-alien.net> References: <47ae4e8ed445c7e6fc46796d46b9a928.authenticated@ultimatedns.net> <20140226000421.GA80747@lor.one-eyed-alien.net> Message-ID: > On Tue, Feb 25, 2014 at 03:31:56PM -0800, Chris H wrote: >> Greetings, >> I'm in the process of performing an upgrade, and ran a preliminary check >> to see what would be performed. During the process I encountered a >> dialog(1) for both clang, and llvm3. It was for textproc/clucene. Which >> wants devel/boost-libs, which depends on devel/boost-jam. I didn't require >> clang, or llvm3 with the initial install of textproc/clucene (that I can see/recall). >> But am presented with the dialog this time. I have WITHOUT_CLANG=true >> in /etc/src.conf. Do I need to also add it to /etc/make.conf? Anything else? > > It looks like this is because boots requires a C++11 compiler and the > default C++11 compiler is clang. From reading the source I deduce that > FAVORITE_COMPILER=gcc in your make.conf may achieve the effect you > require. You rock! I Really appreciate your taking the time to help. --Chris > > -- Brooks > From lev at FreeBSD.org Wed Feb 26 07:02:35 2014 From: lev at FreeBSD.org (Lev Serebryakov) Date: Wed, 26 Feb 2014 11:02:22 +0400 Subject: What is difference between loading module with loader and loading module wtih kldload? In-Reply-To: <9890815.20140226013107@serebryakov.spb.ru> References: <9890815.20140226013107@serebryakov.spb.ru> Message-ID: <1026513823.20140226110222@serebryakov.spb.ru> Hello, Lev. You wrote 26 ??????? 2014 ?., 1:31:07: LS> But if I boot without this module, drop to single-user mode, load module LS> with kldload and continue booting with "exit" everything work smoothly for LS> hours! But then crashes too :( -- // Black Lion AKA Lev Serebryakov From oliver.pntr at gmail.com Wed Feb 26 11:14:06 2014 From: oliver.pntr at gmail.com (Oliver Pinter) Date: Wed, 26 Feb 2014 12:14:05 +0100 Subject: What is difference between loading module with loader and loading module wtih kldload? In-Reply-To: <9890815.20140226013107@serebryakov.spb.ru> References: <9890815.20140226013107@serebryakov.spb.ru> Message-ID: On 2/25/14, Lev Serebryakov wrote: > Hello, Hackers. > > I've upgraded my fileserver to 10-STABLE and got very strange (but > very-very painful) problem, which I could not reproduce on virtual machine > (VirtualBox). > > I'm using geom_raid5 module (and I'm its maintainer, yes) and module, > built > for 10-STABLE (after world & kernel build & install & reboot), and loaded > via /boot/loader.conf is reason to almost instant crash after boot. > Sometimes system mounts filesystems before crash and sometimes not. Most of > time it is "page write: page not present", in different places. PS/2 > keyboard is always blocked after that, I could not drop to debugger. No > memory dump performed. Several times it turend off video output (!) right > after crash. I have a similar crash with geom_sched. This introduced after fine graded geom locks. > > But if I boot without this module, drop to single-user mode, load module > with kldload and continue booting with "exit" everything work smoothly for > hours! > > I understand, that it it some incompatibility between module new kernel, > but I could not reproduce it on VirtualBox instance, and I'm puzzled, that > this crash does not occur if module loaded by kldload! Maybe, here is some > hint in this? > > -- > // Black Lion AKA Lev Serebryakov > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" > From rysto32 at gmail.com Wed Feb 26 19:22:49 2014 From: rysto32 at gmail.com (Ryan Stone) Date: Wed, 26 Feb 2014 14:22:48 -0500 Subject: [PATCH] Add MSI support to puc(9) Message-ID: The Exar XR17V358 is a PCIe device and it supports a single MSI interrupt. This patch will make puc(9) use an MSI interrupt on devices that declare that they support it: http://people.freebsd.org/~rstone/patches/puc_msi.diff This patch may be overly paranoid; I was worried that it's wasn't guaranteed that I could always call pci_alloc_msi() (forgetting that the P in puc stands for PCI) so I added a new puc_cfg_cmd that individual device config methods could implement to advertise support rather than depending on pci_alloc_msi() to behave sanely. I have tested the patch on both a XR17V358 and a XR17V258 (which is a legacy PCI device that does not support PCI) From lukasz.wojcik at zoho.com Wed Feb 26 19:23:49 2014 From: lukasz.wojcik at zoho.com (=?UTF-8?B?xYF1a2FzeiBXw7NqY2lr?=) Date: Wed, 26 Feb 2014 20:05:45 +0100 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: References: Message-ID: <530E3B09.2060602@zoho.com> W dniu 2014-02-25 09:24, Dmitry Selyutin pisze: > Hello everyone! > > My name is Dmitry, I'm 22 years old student from Lomonosov Moscow State > University of Russia. This message is addressed mainly to C connoiseurs, > yet I think other people may find it interesting. It's a GSoC proposal. > It's a pity that for language like C we generally don't have something > universal like Boost, so we have to implement some common functions from > scratch or introduce new dependencies. We have Glib, which is used mainly > by Gnome developers (though it is a standalone library) and LGPL-ed, which > is not as liberal as Boost's license. We also have APR, which seems to be a > bit more comprehensive and convenient, yet it is not so well-known as Glib. > I'm in process of implementing a something like Boost for ANSI C (though I > don't pretend this library to share Boost's comprehensiveness). Several > ideas I find useful are: > > 1. Strict and universal interface. Each function begins with qc_ prefix, > followed by type if function is type-related. Most of types (except > enum-typedef'ed ones) have three common functions (replace _type_ with the > necessary type name, e.g. _bytes_): > a. Constructor: void * qc_type_construct(void). Allocate object instance > and initialize it to default value. > b. Destructor: void qc_type_destruct(void * pointer). Deallocate memory > allocated for object and its members. > c. Replicator: void * qc_type_replicate(void const * pointer). Replicate > object, copying its data to new allocated object, i.e. clone. > Most of types also have _import_ and _export_ functions, which allow to > fill allocated object with the desired data (e.g. fill bytes array from > null-terminated char string) or export data to another type. Almost all > enum-typedef'ed types also have _import_ and _export_ functions to retrieve > a key value corresponding to string. > > 2. Common and universal error type, qc_error. Such errors can be generated > from errno values as well as from Windows API errors. > Almost all functions from qc library except for the three common functions > (constructor, destructor, replicator) return qc_error code and set specific > qc_errno variable to this code. > It is now possible to write such constructions: > if (qc_bytes_import_str(bytes_instance, "Hello, world!")) > qc_errormsg(qc_errno, "error check"); > Global variable qc_errno is implemented in terms of multithreading > applications, it is unique for every running thread. > > 3. Clear distinction between byte and Unicode strings (qc_byte and qc_ucs > types for characters, where qc_byte has exactly 8-bit width and qc_ucs has > exactly 32-bit width). > Charsets are just enumeration, e.g. QC_ENCODING_UTF8, > QC_ENCODING_ISO_8859_4, etc., yet it is possible to lookup the desired > encoding using qc_encoding_import. If encoding is known under several > names, they are handled alltogether, i.e. QC_ENCODING_ASCII will be > returned for any of "ASCII", "iso-ir-6", "ANSI_X3.4-1968", "ISO646-US", > etc. Register doesn't matter. > Two new types, qc_bytes and qc_unicode are provided, in order to store > binary and Unicode data. It is possible to store null characters inside > such objects. It is similar to C++ string and C++ vector classes. > > 4. Universal file system path type. It is possible to achieve > cross-platformness using such path types, i.e. it is possible to make > directories, links, symlinks, remove files, directories, etc. regardless of > platform path API. > > 5. i18n support must be embedded with qc library. Locales, timezones, day > and months names, etc. must be provided too, in several forms if necessary. > i18n functions work with qc_unicode type, so charset conversion problems > won't exist. > > 6. Multithreading support if platform permits it. On POSIX we use pthreads, > on Windows we use its API for multithreading. I'm also thinking about green > threads implementation. Thread local storage is already implemented, yet > there is still a lot work to be done with threads, mutexes, etc. > > 7. Multiprecision arithmetics. I'm using GMP to achieve it, yet I'm > planning to switch to something BSD-like or to implement it from scratch > (that probably would be better, since library doesn't introduce any > dependencies except GMP). > > 8. Ternary logic almost everywhere. Being fan of Russian Setun computer, > I've started this library as implementation of ternary logic operations. > When I've realized that there is still much to be done in other fields, > I've already concluded that ternary logic is suitable for a wide set of > other operations, e.g. comparison, determining type of strings, endianness, > etc. I think that I'll leave type qc_trit, since I've found it extremely > useful. > > 9. A lot of other things to be done, such as unified I/O streams which > provide compression/transformation filters, protocols, math library, etc. > > > Why I'm writing such letter to FreeBSD? I'm pretty sure that FreeBSD highly > needs such library, since we try to use BSD-like license and yet to > implement things that exist in GNU world. The most common example is GNU > readline library, yet I think there is a lot of libraries that we > reimplemented just to let some projects suit BSD philosophy. We can make a > good library, which can be used a lot in different projects, yet it will > field everyone like Boost. The nature of C language itself helps us to keep > this library much more tiny than Boost library; I also think that we can > limit our support with POSIX-(almost-)compatible platforms and Windows. > > I've already done a lot of work, but it is hard to make decisions without > advice or discussion, so I've thought about to enter GSoC this year like I > did two years ago when I've reimplemented gnulib-tool for FSF. This project > is much more interesting for me, since I want to upgrade my C skills, get > known BSD world better and provide a good cross-platform library which may > be useful to almost everyone. Since I've already finished my volunteer's > job in Sochi (I've worked as technologist volunteer), I'm really want to > make something new for other people to be useful. > > If you are interested in my project, here is repository at GitHub: > https://github.com/ghostmansd/quirinus. I guess I'll later rename it to > https://github.com/ghostmansd/qc, yet it doesn't matters. If you're > interested in my previous GSoC work, you can find it here: > https://github.com/ghostmansd/gnulib-python. If you want to get to know me > better from any side (my philological studies in Lomonosov Moscow > University, GSoC participation, Sochi volunteer job, etc.), you can write > me a email; I'd be very glad. > > Thanks for reading such a long letter! > Hi Dmitry, I'd like to say I personally like the idea of having glibc counterpart in FreeBSD (and possibly other OSes). It's not quite sure though whether you're looking for help with development or review ? Perhaps with something else ? Sincerely, -?W From fernando.apesteguia at gmail.com Wed Feb 26 20:02:02 2014 From: fernando.apesteguia at gmail.com (=?ISO-8859-1?Q?Fernando_Apestegu=EDa?=) Date: Wed, 26 Feb 2014 21:02:00 +0100 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: <530E3B09.2060602@zoho.com> References: <530E3B09.2060602@zoho.com> Message-ID: On Wed, Feb 26, 2014 at 8:05 PM, ?ukasz W?jcik wrote: > W dniu 2014-02-25 09:24, Dmitry Selyutin pisze: > > Hello everyone! >> >> My name is Dmitry, I'm 22 years old student from Lomonosov Moscow State >> University of Russia. This message is addressed mainly to C connoiseurs, >> yet I think other people may find it interesting. It's a GSoC proposal. >> It's a pity that for language like C we generally don't have something >> universal like Boost, so we have to implement some common functions from >> scratch or introduce new dependencies. We have Glib, which is used mainly >> by Gnome developers (though it is a standalone library) and LGPL-ed, which >> is not as liberal as Boost's license. We also have APR, which seems to be >> a >> bit more comprehensive and convenient, yet it is not so well-known as >> Glib. >> I'm in process of implementing a something like Boost for ANSI C (though I >> don't pretend this library to share Boost's comprehensiveness). Several >> ideas I find useful are: >> >> 1. Strict and universal interface. Each function begins with qc_ prefix, >> followed by type if function is type-related. Most of types (except >> enum-typedef'ed ones) have three common functions (replace _type_ with the >> necessary type name, e.g. _bytes_): >> a. Constructor: void * qc_type_construct(void). Allocate object >> instance >> and initialize it to default value. >> b. Destructor: void qc_type_destruct(void * pointer). Deallocate memory >> allocated for object and its members. >> c. Replicator: void * qc_type_replicate(void const * pointer). >> Replicate >> object, copying its data to new allocated object, i.e. clone. >> Most of types also have _import_ and _export_ functions, which allow to >> fill allocated object with the desired data (e.g. fill bytes array from >> null-terminated char string) or export data to another type. Almost all >> enum-typedef'ed types also have _import_ and _export_ functions to >> retrieve >> a key value corresponding to string. >> >> 2. Common and universal error type, qc_error. Such errors can be generated >> from errno values as well as from Windows API errors. >> Almost all functions from qc library except for the three common functions >> (constructor, destructor, replicator) return qc_error code and set >> specific >> qc_errno variable to this code. >> It is now possible to write such constructions: >> if (qc_bytes_import_str(bytes_instance, "Hello, world!")) >> qc_errormsg(qc_errno, "error check"); >> Global variable qc_errno is implemented in terms of multithreading >> applications, it is unique for every running thread. >> >> 3. Clear distinction between byte and Unicode strings (qc_byte and qc_ucs >> types for characters, where qc_byte has exactly 8-bit width and qc_ucs has >> exactly 32-bit width). >> Charsets are just enumeration, e.g. QC_ENCODING_UTF8, >> QC_ENCODING_ISO_8859_4, etc., yet it is possible to lookup the desired >> encoding using qc_encoding_import. If encoding is known under several >> names, they are handled alltogether, i.e. QC_ENCODING_ASCII will be >> returned for any of "ASCII", "iso-ir-6", "ANSI_X3.4-1968", "ISO646-US", >> etc. Register doesn't matter. >> Two new types, qc_bytes and qc_unicode are provided, in order to store >> binary and Unicode data. It is possible to store null characters inside >> such objects. It is similar to C++ string and C++ vector classes. >> >> 4. Universal file system path type. It is possible to achieve >> cross-platformness using such path types, i.e. it is possible to make >> directories, links, symlinks, remove files, directories, etc. regardless >> of >> platform path API. >> >> 5. i18n support must be embedded with qc library. Locales, timezones, day >> and months names, etc. must be provided too, in several forms if >> necessary. >> i18n functions work with qc_unicode type, so charset conversion problems >> won't exist. >> >> 6. Multithreading support if platform permits it. On POSIX we use >> pthreads, >> on Windows we use its API for multithreading. I'm also thinking about >> green >> threads implementation. Thread local storage is already implemented, yet >> there is still a lot work to be done with threads, mutexes, etc. >> >> 7. Multiprecision arithmetics. I'm using GMP to achieve it, yet I'm >> planning to switch to something BSD-like or to implement it from scratch >> (that probably would be better, since library doesn't introduce any >> dependencies except GMP). >> >> 8. Ternary logic almost everywhere. Being fan of Russian Setun computer, >> I've started this library as implementation of ternary logic operations. >> When I've realized that there is still much to be done in other fields, >> I've already concluded that ternary logic is suitable for a wide set of >> other operations, e.g. comparison, determining type of strings, >> endianness, >> etc. I think that I'll leave type qc_trit, since I've found it extremely >> useful. >> >> 9. A lot of other things to be done, such as unified I/O streams which >> provide compression/transformation filters, protocols, math library, etc. >> >> >> Why I'm writing such letter to FreeBSD? I'm pretty sure that FreeBSD >> highly >> needs such library, since we try to use BSD-like license and yet to >> implement things that exist in GNU world. The most common example is GNU >> readline library, yet I think there is a lot of libraries that we >> reimplemented just to let some projects suit BSD philosophy. We can make a >> good library, which can be used a lot in different projects, yet it will >> field everyone like Boost. The nature of C language itself helps us to >> keep >> this library much more tiny than Boost library; I also think that we can >> limit our support with POSIX-(almost-)compatible platforms and Windows. >> >> I've already done a lot of work, but it is hard to make decisions without >> advice or discussion, so I've thought about to enter GSoC this year like I >> did two years ago when I've reimplemented gnulib-tool for FSF. This >> project >> is much more interesting for me, since I want to upgrade my C skills, get >> known BSD world better and provide a good cross-platform library which may >> be useful to almost everyone. Since I've already finished my volunteer's >> job in Sochi (I've worked as technologist volunteer), I'm really want to >> make something new for other people to be useful. >> >> If you are interested in my project, here is repository at GitHub: >> https://github.com/ghostmansd/quirinus. I guess I'll later rename it to >> https://github.com/ghostmansd/qc, yet it doesn't matters. If you're >> interested in my previous GSoC work, you can find it here: >> https://github.com/ghostmansd/gnulib-python. If you want to get to know >> me >> better from any side (my philological studies in Lomonosov Moscow >> University, GSoC participation, Sochi volunteer job, etc.), you can write >> me a email; I'd be very glad. >> >> Thanks for reading such a long letter! >> >> > Hi Dmitry, > > I'd like to say I personally like the idea of having glibc counterpart in > FreeBSD (and possibly other OSes). It's not quite sure though whether > you're looking for help with development or review ? Perhaps with something > else ? > I suppose you mean _glib_ instead of glibc (our glibc counterpart is our libc in base). The project sounds nice although a lot of features (like the error treatment, multithreading support...) are already present in glib (and they are very well tested). Cheers. > > Sincerely, > -?W > > > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" > From rysto32 at gmail.com Wed Feb 26 20:33:13 2014 From: rysto32 at gmail.com (Ryan Stone) Date: Wed, 26 Feb 2014 15:33:12 -0500 Subject: [PATCH] Add MSI support to puc(9) In-Reply-To: References: Message-ID: Oops, I just realized that when I pulled the patch from our internal tree and applied it to head that pucdata.c was not patched correctly. I have corrected the patch and uploaded a new version. From wjw at digiware.nl Wed Feb 26 20:34:24 2014 From: wjw at digiware.nl (Willem Jan Withagen) Date: Wed, 26 Feb 2014 21:34:20 +0100 Subject: Variant Link (Was: Re: Thoughts on Multi-Symlink Concept) In-Reply-To: References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> <530BE132.1030507@m5p.com> Message-ID: <530E4FCC.4090401@digiware.nl> On 25-2-2014 2:20, Daniel Eischen wrote: > On Mon, 24 Feb 2014, George Mitchell wrote: >> On 02/22/14 21:45, Jordan Hubbard wrote: >>> On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen >>>> Apollo Domain systems had those, and they were great. >>>> Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or >>>> SYSV to get the other stuff. >>> [...] >> >> Since we're going down DomainOS Memory Lane, does anyone else miss >> transcript pads? -- George > > I missed the transcript pad too. > > I also missed the DM editor and the ability to write DM scripts. > > And I miss the edit pad - the first time I was able to do > a rectangular cut/paste/move. And I loved the ability to place > the cursor beyond the current end of a line and start typing > (without clicking the mouse!). I hate having to hold down > the space bar, adding blanks, to get out to the column I want > (suppose your coding conventions disallow tabs). If I remember correctly it was possible to run X on the box, in a sort of transparent thru mode. And that was version 10.?? But first time I saw it, I wondered why anybody would want to run that on DomainOS? It was progress the wrong way around.... And from the DM editor, I went into vi, because Emacs just does not fit in the frame of my brain.... :( BTW Got an svn URL from Brooks@, so I'll atleast download the VariantLink code.... --WjW From jkh at turbofuzz.com Wed Feb 26 21:01:45 2014 From: jkh at turbofuzz.com (Jordan Hubbard) Date: Wed, 26 Feb 2014 12:55:11 -0800 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: References: Message-ID: Hi Dmitry, Let me first just say that this is a great idea - libc has always traditionally been a very minimalist place to start writing applications from, and certainly ?lack of high-level API? is one of the traditional Achilles heels of Unix, not just FreeBSD. I can certainly say from personal experience that the life of an ISV on a Unix platform is a sad one - you either end up writing a lot of the foundational libraries yourself and/or you stick together a potpourri of libraries like apr, glib and gmp to try and give your application a somewhat higher place to stand, hoping like hell of course that the foundational libraries you?ve chosen actually work, and work together! That said, I feel compelled to point out that you?re also about to drive into a swamp, and there are lots of alligators and poisonous snakes in there. :-) If you?re going to do this, it?s best to go into it with your eyes fully open and aware of the magnitude of the challenge you?re about to undertake (hint - you will be at least 28 years old before you consider this ?done? enough to walk away, or more likely run screaming, from it because by then you?ll be really really ready to move onto something else!). I don?t say this to discourage you, I just want to make sure you?re fully aware of what you?re about to undertake if you?re truly committed to seeing it all the way through - this is far more than just a GSoC project! One of the big reasons it?s such a swamp is the degree to which things quickly become interconnected inside such a library. You want everything to be I18N compliant by default, so first you implement the Unicode character handling support and the relevant character classing functions, and of course Unicode also means strings, so you implement your own unicode-aware String type and also add a bunch of convenience functions to Strings while you?re in there. Then, naturally, you need to convert other data types (like C strings) to and from your String type, so you add those functions (and if you want to see how far that can go - look here!). Once you?ve done all that work, you want the rest of your library to use your String type, so now you?ve got your second (after the character handling) bit of inter-connectedness and it just multiplies from there. All your base types (Strings, Hashes, Arrays, Trees, etc) become consumed by the rest of the library and any client of gc_lib soon needs to sign up for the whole package - they can?t just pick and choose individual functions as they see fit, and maybe that?s OK, but that?s part of the reason why there are so many competing libraries like this - everyone wants to own the foundation because it?s just so much easier when they do! Perhaps all of this has already occurred to you, in which case I?m just stating the obvious, but having written such libraries myself (libxtr - stood for ?extra functions? - back in the 1990?s) and also having had recent experience with Apple?s Foundation libraries, which do everything you just described and a lot more, I figured it wouldn?t hurt to point it out. Anyway, on to the specifics: On Feb 25, 2014, at 12:24 AM, Dmitry Selyutin wrote: > 1. Strict and universal interface. Each function begins with qc_ prefix, > followed by type if function is type-related. Yep, it?s good you put that first. If the names don?t make sense, nobody can remember what to call and the library is useless! Uniform calling conventions for everything is also obviously important. > 4. Universal file system path type. It is possible to achieve > cross-platformness using such path types, i.e. it is possible to make > directories, links, symlinks, remove files, directories, etc. regardless of > platform path API. Here, however, I think you might want to really consider the role of this library. Is it a ?higher level library for BSD? or is it a cross-platform development library that caters to non-Unix platforms as well? You mentioned Windows earlier in the document, and if so then that?s cause for concern because if you really want to make it a cross-platform library then you?re going to be faced with lots of different constraints that just don?t apply to FreeBSD and will seriously limit the approaches you can take. Moreover, the code itself will be a lot uglier if it has to compile on Windows (or $someOtherNonUnixOS) as well, and you will spend a substantial amount of time and energy just impedance matching the library to the different OS foundations. Personally, I think it should be a non-goal to make this library cross-platform. There are already plenty of other good cross-platform development libraries out there if that?s what an ISV wants to do, whereas there is no ?better libc than libc? library for *BSD, which holds native development back. More importantly, if you just focus on BSD as your foundation, you can make a lot of assumptions and leverage the power of the underlying OS in ways you just won?t be able to do if you want this to also work on Windows. > 6. Multithreading support if platform permits it. On POSIX we use pthreads, > on Windows we use its API for multithreading. I'm also thinking about green > threads implementation. Thread local storage is already implemented, yet > there is still a lot work to be done with threads, mutexes, etc. Please don?t reinvent pthreads. They already suck enough in libc. :-) I?ve been up on the libdispatch soapbox so many times that I?m sure people here are tired of hearing it, but seriously, once Grand Central Dispatch became firmly embedded in iOS and OS X, the app writers never looked back at pthreads again because GCD is just so much easier to use, and takes so much of the pain out of parallelism, that I?ve received countless emails from developers crying that they were recently forced to go back to pthreads on some other platform and had forgotten how horrible and painful the experience was! GCD has also been ported to Windows, Linux and Android (again, by some of those same developers who just would not, could not, go back to pthreads on Android) largely on the strength of how much better an API it is for multi-threaded programming. It?s also been ported to FreeBSD (Hi Robert) so there?s really just no need to re-invent this wheel. The source code is there, it works, just make it a subset of your library. Its dependencies are deliberately very small because we wanted it to be low-level and cross-platform. > 9. A lot of other things to be done, such as unified I/O streams which > provide compression/transformation filters, protocols, math library, etc. Look at GCD?s async I/O functions and, if you want to create transformation/compression filters, you really can?t do better than Security Transforms as a model. Don?t let the name fool you - they were created for cryptographic tasks, but they?re really a generic, multithreaded, data pipelining model. If I had those to do over again, I would not have had them called SecTransforms at all but merely added a Transform API to GCD and then built SecTransforms on top! The rest of your proposal looks fine to me - date functions, unicode, etc - all pretty standard stuff. Like I said, I have pretty recent experience with this with Core Foundation (see also OpenCFLite) and it?s all doable and very useful, you just need to remember all the caveats in my first paragraph! :-) - Jordan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 236 bytes Desc: Message signed with OpenPGP using GPGMail URL: From deischen at freebsd.org Wed Feb 26 21:21:55 2014 From: deischen at freebsd.org (Daniel Eischen) Date: Wed, 26 Feb 2014 16:21:44 -0500 (EST) Subject: Variant Link (Was: Re: Thoughts on Multi-Symlink Concept) In-Reply-To: <530E4FCC.4090401@digiware.nl> References: <530049a1.XXZ1PjZFgRyCu9X6%perryh@pluto.rain.com> <53092D83.6050603@digiware.nl> <43505B61-FAE8-4A61-922E-78F6007BBFC3@gmail.com> <530BE132.1030507@m5p.com> <530E4FCC.4090401@digiware.nl> Message-ID: On Wed, 26 Feb 2014, Willem Jan Withagen wrote: > On 25-2-2014 2:20, Daniel Eischen wrote: >> On Mon, 24 Feb 2014, George Mitchell wrote: >>> On 02/22/14 21:45, Jordan Hubbard wrote: >>>> On Feb 22, 2014, at 3:06 PM, Willem Jan Withagen >>>>> Apollo Domain systems had those, and they were great. >>>>> Set SYSTYPE to BSD4 and get the BSD tree and all that came with it, or >>>>> SYSV to get the other stuff. >>>> [...] >>> >>> Since we're going down DomainOS Memory Lane, does anyone else miss >>> transcript pads? -- George >> >> I missed the transcript pad too. >> >> I also missed the DM editor and the ability to write DM scripts. >> >> And I miss the edit pad - the first time I was able to do >> a rectangular cut/paste/move. And I loved the ability to place >> the cursor beyond the current end of a line and start typing >> (without clicking the mouse!). I hate having to hold down >> the space bar, adding blanks, to get out to the column I want >> (suppose your coding conventions disallow tabs). > > If I remember correctly it was possible to run X on the box, in a sort of > transparent thru mode. And that was version 10.?? > > But first time I saw it, I wondered why anybody would want to run that > on DomainOS? It was progress the wrong way around.... tank (or perhaps it is xtank now?) was awesome on the DN10000. My memory is foggy - I forget some of the other games. > And from the DM editor, I went into vi, because Emacs just does not fit > in the frame of my brain.... :( > > BTW Got an svn URL from Brooks@, so I'll atleast download the > VariantLink code.... Cool. If we were able to commit this, we wouldn't have to have Domain/OS conversations every 7 years :( -- DE From noname.esst at yahoo.com Thu Feb 27 06:22:43 2014 From: noname.esst at yahoo.com (Nomad Esst) Date: Wed, 26 Feb 2014 22:19:49 -0800 (PST) Subject: Writting eeprom Message-ID: <1393481989.22330.YahooMailNeo@web162704.mail.bf1.yahoo.com> We want to add our own ID for each of our modules (pci cards), is it possible to write this ID on somewhere of the card like EEPROM? Thanks in advance From lukasz.wojcik at zoho.com Thu Feb 27 09:34:34 2014 From: lukasz.wojcik at zoho.com (=?ISO-8859-2?Q?=A3ukasz_W=F3jcik?=) Date: Thu, 27 Feb 2014 10:34:31 +0100 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: References: <530E3B09.2060602@zoho.com> Message-ID: <530F06A7.3090005@zoho.com> W dniu 2014-02-26 21:02, Fernando Apestegu?a pisze: > On Wed, Feb 26, 2014 at 8:05 PM, ?ukasz W?jcik wrote: > >> W dniu 2014-02-25 09:24, Dmitry Selyutin pisze: >> >> Hello everyone! >>> >>> My name is Dmitry, I'm 22 years old student from Lomonosov Moscow State >>> University of Russia. This message is addressed mainly to C connoiseurs, >>> yet I think other people may find it interesting. It's a GSoC proposal. >>> It's a pity that for language like C we generally don't have something >>> universal like Boost, so we have to implement some common functions from >>> scratch or introduce new dependencies. We have Glib, which is used mainly >>> by Gnome developers (though it is a standalone library) and LGPL-ed, which >>> is not as liberal as Boost's license. We also have APR, which seems to be >>> a >>> bit more comprehensive and convenient, yet it is not so well-known as >>> Glib. >>> I'm in process of implementing a something like Boost for ANSI C (though I >>> don't pretend this library to share Boost's comprehensiveness). Several >>> ideas I find useful are: >>> >>> 1. Strict and universal interface. Each function begins with qc_ prefix, >>> followed by type if function is type-related. Most of types (except >>> enum-typedef'ed ones) have three common functions (replace _type_ with the >>> necessary type name, e.g. _bytes_): >>> a. Constructor: void * qc_type_construct(void). Allocate object >>> instance >>> and initialize it to default value. >>> b. Destructor: void qc_type_destruct(void * pointer). Deallocate memory >>> allocated for object and its members. >>> c. Replicator: void * qc_type_replicate(void const * pointer). >>> Replicate >>> object, copying its data to new allocated object, i.e. clone. >>> Most of types also have _import_ and _export_ functions, which allow to >>> fill allocated object with the desired data (e.g. fill bytes array from >>> null-terminated char string) or export data to another type. Almost all >>> enum-typedef'ed types also have _import_ and _export_ functions to >>> retrieve >>> a key value corresponding to string. >>> >>> 2. Common and universal error type, qc_error. Such errors can be generated >>> from errno values as well as from Windows API errors. >>> Almost all functions from qc library except for the three common functions >>> (constructor, destructor, replicator) return qc_error code and set >>> specific >>> qc_errno variable to this code. >>> It is now possible to write such constructions: >>> if (qc_bytes_import_str(bytes_instance, "Hello, world!")) >>> qc_errormsg(qc_errno, "error check"); >>> Global variable qc_errno is implemented in terms of multithreading >>> applications, it is unique for every running thread. >>> >>> 3. Clear distinction between byte and Unicode strings (qc_byte and qc_ucs >>> types for characters, where qc_byte has exactly 8-bit width and qc_ucs has >>> exactly 32-bit width). >>> Charsets are just enumeration, e.g. QC_ENCODING_UTF8, >>> QC_ENCODING_ISO_8859_4, etc., yet it is possible to lookup the desired >>> encoding using qc_encoding_import. If encoding is known under several >>> names, they are handled alltogether, i.e. QC_ENCODING_ASCII will be >>> returned for any of "ASCII", "iso-ir-6", "ANSI_X3.4-1968", "ISO646-US", >>> etc. Register doesn't matter. >>> Two new types, qc_bytes and qc_unicode are provided, in order to store >>> binary and Unicode data. It is possible to store null characters inside >>> such objects. It is similar to C++ string and C++ vector classes. >>> >>> 4. Universal file system path type. It is possible to achieve >>> cross-platformness using such path types, i.e. it is possible to make >>> directories, links, symlinks, remove files, directories, etc. regardless >>> of >>> platform path API. >>> >>> 5. i18n support must be embedded with qc library. Locales, timezones, day >>> and months names, etc. must be provided too, in several forms if >>> necessary. >>> i18n functions work with qc_unicode type, so charset conversion problems >>> won't exist. >>> >>> 6. Multithreading support if platform permits it. On POSIX we use >>> pthreads, >>> on Windows we use its API for multithreading. I'm also thinking about >>> green >>> threads implementation. Thread local storage is already implemented, yet >>> there is still a lot work to be done with threads, mutexes, etc. >>> >>> 7. Multiprecision arithmetics. I'm using GMP to achieve it, yet I'm >>> planning to switch to something BSD-like or to implement it from scratch >>> (that probably would be better, since library doesn't introduce any >>> dependencies except GMP). >>> >>> 8. Ternary logic almost everywhere. Being fan of Russian Setun computer, >>> I've started this library as implementation of ternary logic operations. >>> When I've realized that there is still much to be done in other fields, >>> I've already concluded that ternary logic is suitable for a wide set of >>> other operations, e.g. comparison, determining type of strings, >>> endianness, >>> etc. I think that I'll leave type qc_trit, since I've found it extremely >>> useful. >>> >>> 9. A lot of other things to be done, such as unified I/O streams which >>> provide compression/transformation filters, protocols, math library, etc. >>> >>> >>> Why I'm writing such letter to FreeBSD? I'm pretty sure that FreeBSD >>> highly >>> needs such library, since we try to use BSD-like license and yet to >>> implement things that exist in GNU world. The most common example is GNU >>> readline library, yet I think there is a lot of libraries that we >>> reimplemented just to let some projects suit BSD philosophy. We can make a >>> good library, which can be used a lot in different projects, yet it will >>> field everyone like Boost. The nature of C language itself helps us to >>> keep >>> this library much more tiny than Boost library; I also think that we can >>> limit our support with POSIX-(almost-)compatible platforms and Windows. >>> >>> I've already done a lot of work, but it is hard to make decisions without >>> advice or discussion, so I've thought about to enter GSoC this year like I >>> did two years ago when I've reimplemented gnulib-tool for FSF. This >>> project >>> is much more interesting for me, since I want to upgrade my C skills, get >>> known BSD world better and provide a good cross-platform library which may >>> be useful to almost everyone. Since I've already finished my volunteer's >>> job in Sochi (I've worked as technologist volunteer), I'm really want to >>> make something new for other people to be useful. >>> >>> If you are interested in my project, here is repository at GitHub: >>> https://github.com/ghostmansd/quirinus. I guess I'll later rename it to >>> https://github.com/ghostmansd/qc, yet it doesn't matters. If you're >>> interested in my previous GSoC work, you can find it here: >>> https://github.com/ghostmansd/gnulib-python. If you want to get to know >>> me >>> better from any side (my philological studies in Lomonosov Moscow >>> University, GSoC participation, Sochi volunteer job, etc.), you can write >>> me a email; I'd be very glad. >>> >>> Thanks for reading such a long letter! >>> >>> >> Hi Dmitry, >> >> I'd like to say I personally like the idea of having glibc counterpart in >> FreeBSD (and possibly other OSes). It's not quite sure though whether >> you're looking for help with development or review ? Perhaps with something >> else ? >> > > I suppose you mean _glib_ instead of glibc (our glibc counterpart is our > libc in base). > > The project sounds nice although a lot of features (like the error > treatment, multithreading support...) are already present in glib (and they > are very well tested). > > Cheers. Yes, of course, I meant glib. As you know it's much much more feature rich than standard libc, e.g. xml parsing, regular expressions, key-value parsers and many, many more. > > >> >> Sincerely, >> -?W >> >> >> _______________________________________________ >> freebsd-hackers at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" >> > _______________________________________________ > freebsd-hackers at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org" > From ghostman.sd at gmail.com Thu Feb 27 15:39:47 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Thu, 27 Feb 2014 19:39:45 +0400 Subject: GSoC proposal: Quirinus C library (qc) Message-ID: Hi Jordan, ?ukasz, Fernando, thanks for your kind replies! There is a lot of questions, I will try to answer everything, if I miss anything, just let me know. :-) I will start with some fixes proposed by ?ukasz. If you look into src/core/error/posix.c, you will find exactly what you have talked about (not sure if I have already pushed it though). I am going to switch to such style everywhere, that is just a matter of time. And yes, you are right with variable name :-) To Fernando: yes, I know about error treatment. The main goal of qc_error type is to provide more descriptive error codes than we have (e.g. we have EILSEQ, but that is not enough to know the real error in Unicode handling), also I wanted to provide a conversion algorithm that aims to make a mapping between POSIX and Windows (errno vs GetLastError). Now it is time for a long response to Jordan. First of all, I realize comprehensiveness of this project: no one will write Boost in three months. :-) Of course that will be just the beginning. The reason why I decided to try and join GSoC with this project is that I thought it may be to find someone who understands better both C and FreeBSD needs to guide me. I need a wise and experienced mentor who may give good advice or help me to select right way. As for the portability in terms of modules, I am planning to separate the project into modules such a qc/core.h|-lqc_core, qc/codecs.h|-lqc_codecs, etc. Of course I am afraid that sooner or later one module may require another one (e.g. file streams may require qcsys module, which contains qc_path-related functions); I am trying to avoid such problems even if sometimes it requires code duplication. You may see endianness detection inside qc_codecs_u16_decode, where I have stated that I would rather detect endianness here rather that call qc_sys_byteorder. I know that we must avoid code duplication; I think it is where we have to choose from two evils. As for strings, I will not use UTF-16 since it provides more problems rather than solutions. If I provide a function which accepts char* or char const* argument, I imply that such function uses only ASCII (may be I will change ASCII to UTF-8). Encoding is used only if a user has requested it explicitly; the only place where I have made exception is system path since path requires to be in UTF-16 on Windows. That is the reason why qc_path requires qc_codecs-related functions. i18n is one of the most complicated parts and is one of the main reasons why I have made a decision to write this library. We have ICU, which is ? just in my opinion ? an ugly, dead-born monster mainly due to mix of C, C++ and Java concepts. It is well tested though, so we can just take all the best from it and never touch the rest. Since I have always thought that deeds speak louder than words (facta sunt potentiora uerbis). And I am pretty sure that good C library which was made by FreeBSD may persuade someone to migrate to it from Windows as it sometimes happens with people who have got addicted to bash. :-) At least, even if we drop Windows support (it seems, however, to be a bad idea to me), I think that we must support POSIX-compliant systems at least. As for GCD, I will think about it. That is the situation where we need to choose between new dependancy and reinventing a new wheel. Honestly, I have always thought that threads suck; I guess if this concept had been created nowadays, it would have not been such a bullshit. Thank you all for your replies again, especially to Jordan. If I missed anything, please remind me. Looking forward to your replies! Best regards, Dmitry Selyutin From jmg at funkthat.com Thu Feb 27 18:15:49 2014 From: jmg at funkthat.com (John-Mark Gurney) Date: Thu, 27 Feb 2014 10:15:47 -0800 Subject: Writting eeprom In-Reply-To: <1393481989.22330.YahooMailNeo@web162704.mail.bf1.yahoo.com> References: <1393481989.22330.YahooMailNeo@web162704.mail.bf1.yahoo.com> Message-ID: <20140227181547.GD47921@funkthat.com> Nomad Esst wrote this message on Wed, Feb 26, 2014 at 22:19 -0800: > We want to add our own ID for each of our modules (pci cards), is it possible to write this ID on somewhere of the card like EEPROM? You need to be more specific in your question. Some pci cards do not have any EEPROM or programable memory on them, so it would be imposible to write to them... with out know which pci cards you are talking about it is not possible to answer your question.. You may have sent this to the wrong list since we don't know the context you are refering to.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From jmg at funkthat.com Thu Feb 27 18:26:43 2014 From: jmg at funkthat.com (John-Mark Gurney) Date: Thu, 27 Feb 2014 10:26:41 -0800 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: References: Message-ID: <20140227182641.GE47921@funkthat.com> Dmitry Selyutin wrote this message on Thu, Feb 27, 2014 at 19:39 +0400: > As for strings, I will not use UTF-16 since it provides more problems > rather than solutions. If I provide a function which accepts char* or char > const* argument, I imply that such function uses only ASCII (may be I will > change ASCII to UTF-8). Encoding is used only if a user has requested it > explicitly; the only place where I have made exception is system path since > path requires to be in UTF-16 on Windows. That is the reason why qc_path > requires qc_codecs-related functions. You do realize that FreeBSD does not enforce any coding on path names current, correct? So, requiring a coding format on FreeBSD (UTF-16) will mean some paths may not be accessible, since I assume you conver the UTF-16 string to UTF-8 before opening on FreeBSD... Hmm.. maybe it's time for a sysctl you can set on your system that only allows you to create UTF-8 valid names to allow people to slowly migrate to UTF-8? and a tool to report/convert old non-UTF-8 paths? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From ghostman.sd at gmail.com Thu Feb 27 19:42:18 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Thu, 27 Feb 2014 23:41:56 +0400 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: References: <20140227182641.GE47921@funkthat.com> Message-ID: Ah, yes, I've misunderstood your question. Yes, on POSIX systems default encoding is taken from setlocale. Then qc_encoding_import converts encoding name to qc_encoding type or leaves it ASCII if it is impossible to find the desired encoding. So conversion from qc_unicode to qc_path is performed using default system encoding. On Windows thinks behave other way though. To summarize: POSIX: qc_path_import_str: copy characters from string to qc_path buffer. qc_path_import_wstr: convert wide string to qc_unicode, decode using locale encoding, copy raw bytes. qc_path_import_bytes: copy raw bytes to qc_path buffer. qc_path_import_unicode: decode using locale encoding, copy raw bytes. Windows: qc_path_import_str: convert to qc_bytes, call qc_path_import_bytes. qc_path_import_wstr: copy wide characters to qc_path buffer. qc_path_import_bytes: decode using UTF-8 encoding, convert to wide string, copy wide string to qc_path buffer. qc_path_import_unicode: convert to wide string, copy wide string char-by-char to qc_path buffer. This is how it works now, though I guess some details may be changed in future. 2014-02-27 22:48 GMT+04:00 Dmitry Selyutin : > Hi John-Mark. > > it seems I've stated things wrong or you've understood me incorrectly. :-) > Path will be in UTF-16 on Windows, otherwise it has pure bytes form, since > paths on POSIX are just sequence of bytes. AFAIK we can use UTF-8 for sure > in OS X though. > So, we have qc_byte for raw bytes (qc_byte == uint8_t), qc_ucs for Unicode > characters (qc_ucs == uint32_t), qc_bytes for raw byte strings, qc_unicode > for Unicode strings. Things get more compicated with paths though. Really > qc_path just stores void* pointer to byte array, which is UTF-16LE sequence > on Windows and raw byte sequence on other platforms. That opens a way to > write a set of platfrom-agnostic functions both on POSIX and Windows. > > > 2014-02-27 22:26 GMT+04:00 John-Mark Gurney : > > Dmitry Selyutin wrote this message on Thu, Feb 27, 2014 at 19:39 +0400: >> > As for strings, I will not use UTF-16 since it provides more problems >> > rather than solutions. If I provide a function which accepts char* or >> char >> > const* argument, I imply that such function uses only ASCII (may be I >> will >> > change ASCII to UTF-8). Encoding is used only if a user has requested it >> > explicitly; the only place where I have made exception is system path >> since >> > path requires to be in UTF-16 on Windows. That is the reason why qc_path >> > requires qc_codecs-related functions. >> >> You do realize that FreeBSD does not enforce any coding on path names >> current, correct? So, requiring a coding format on FreeBSD (UTF-16) >> will mean some paths may not be accessible, since I assume you conver >> the UTF-16 string to UTF-8 before opening on FreeBSD... >> >> Hmm.. maybe it's time for a sysctl you can set on your system that >> only allows you to create UTF-8 valid names to allow people to slowly >> migrate to UTF-8? and a tool to report/convert old non-UTF-8 paths? >> >> -- >> John-Mark Gurney Voice: +1 415 225 5579 >> >> "All that I will do, has been done, All that I have, has not." >> > > > > -- > With best regards, > Dmitry Selyutin > -- With best regards, Dmitry Selyutin From jhb at freebsd.org Thu Feb 27 20:09:25 2014 From: jhb at freebsd.org (John Baldwin) Date: Thu, 27 Feb 2014 13:30:02 -0500 Subject: [PATCH] Add MSI support to puc(9) In-Reply-To: References: Message-ID: <201402271330.02699.jhb@freebsd.org> On Wednesday, February 26, 2014 2:22:48 pm Ryan Stone wrote: > The Exar XR17V358 is a PCIe device and it supports a single MSI > interrupt. This patch will make puc(9) use an MSI interrupt on > devices that declare that they support it: > > http://people.freebsd.org/~rstone/patches/puc_msi.diff > > This patch may be overly paranoid; I was worried that it's wasn't > guaranteed that I could always call pci_alloc_msi() (forgetting that > the P in puc stands for PCI) so I added a new puc_cfg_cmd that > individual device config methods could implement to advertise support > rather than depending on pci_alloc_msi() to behave sanely. > > I have tested the patch on both a XR17V358 and a XR17V258 (which is a > legacy PCI device that does not support PCI) I would suggest reworking this so that you try MSI for all PCI devices. I would do this by removing the 'sc_irid = 0' from puc_bfe_attach() so that it can be set by callers. You could then add attach/detach routines in puc_pci.c that use pci_alloc_msi() and set sc_irid to 1 if MSI works. The sc_irid value would also work as a flag for knowing if detach needs to call pci_release_msi() (or puc_pci_attach() handling failure in puc_bfe_attach()) (though I wouldn't be opposed to keeping sc_msi as a separate flag). -- John Baldwin From lichray at gmail.com Thu Feb 27 21:09:52 2014 From: lichray at gmail.com (Zhihao Yuan) Date: Thu, 27 Feb 2014 16:09:51 -0500 Subject: Please update ncurses in base Message-ID: libncurses in base handles wide characters inconsistently and behaves completely broken under some locales (zh_CN.GB2312, zh_CN.GBK, ko_KR.CP949). Those bugs are fixed in 20081129 and the unctrl (which usually mis-escape wide characters) have been rework by the end of 2009. The ncurses 5.9 in ports works much better then the one in base, and is actively fixing bugs on BSDs, why not import this library? -- Zhihao Yuan, ID lichray The best way to predict the future is to invent it. ___________________________________________________ 4BSD -- http://4bsd.biz/ From mail.jiang.cn at gmail.com Fri Feb 28 03:06:53 2014 From: mail.jiang.cn at gmail.com (=?UTF-8?B?5pyx5rGf?=) Date: Fri, 28 Feb 2014 11:06:51 +0800 Subject: Anyone interested in Atom Editor? Message-ID: Hi guys, I have one invite for Atom Editor , is there anyone interested in Atom Editor and need my invitation? If you need this invitation, please let me know your e-mail. BTW, when you successfully be invited, you got 3 invites. I hope who get my invite should invite other people who need invitation as well. Best regards, -- Jiang Zhu mail.jiang.cn at gmail.com From mail.jiang.cn at gmail.com Fri Feb 28 04:09:49 2014 From: mail.jiang.cn at gmail.com (=?UTF-8?B?5pyx5rGf?=) Date: Fri, 28 Feb 2014 12:09:47 +0800 Subject: Anyone interested in Atom Editor? In-Reply-To: References: Message-ID: 2014-02-28 11:06 GMT+08:00 ?? : > > Hi guys, > > I have one invite for Atom Editor , is there anyone > interested in Atom Editor and need my invitation? If you need this > invitation, please let me know your e-mail. > > BTW, when you successfully be invited, you got 3 invites. I hope who get > my invite should invite other people who need invitation as well. > > Best regards, > -- > Jiang Zhu > mail.jiang.cn at gmail.com > Hi guys, My invitation have sent to Chad Milios at ccsys.com, maybe you can ask Chad to share with you. Best regards, -- Jiang Zhu mail.jiang.cn at gmail.com From kevin.bowling at kev009.com Fri Feb 28 18:51:41 2014 From: kevin.bowling at kev009.com (Kevin Bowling) Date: Fri, 28 Feb 2014 11:51:20 -0700 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: References: Message-ID: On 2/25/2014 1:24 AM, Dmitry Selyutin wrote: > Hello everyone! > > My name is Dmitry, I'm 22 years old student from Lomonosov Moscow State > University of Russia. This message is addressed mainly to C connoiseurs, > yet I think other people may find it interesting. It's a GSoC proposal. > It's a pity that for language like C we generally don't have something > universal like Boost, so we have to implement some common functions from > scratch or introduce new dependencies. We have Glib, which is used mainly > by Gnome developers (though it is a standalone library) and LGPL-ed, which > is not as liberal as Boost's license. We also have APR, which seems to be a > bit more comprehensive and convenient, yet it is not so well-known as Glib. > I'm in process of implementing a something like Boost for ANSI C (though I > don't pretend this library to share Boost's comprehensiveness). Several > ideas I find useful are: Depending on what kind of development you are doing, either of these make a decent foundation library: * http://facebook.github.io/libphenom/ * https://github.com/joyent/libuv You may want to pick and use one wholesale for any I/O pieces. Regards, Kevin From jkh at mail.turbofuzz.com Fri Feb 28 19:20:40 2014 From: jkh at mail.turbofuzz.com (Jordan Hubbard) Date: Fri, 28 Feb 2014 11:20:37 -0800 Subject: GSoC proposal: Quirinus C library (qc) In-Reply-To: References: Message-ID: <0AF609C2-A90D-4BFD-8281-A52FB0797A79@mail.turbofuzz.com> It used to be true that you weren't a "Real Programmer?" until you'd written your own screen editor. I guess now it's true that you're not one until you've written your own async event dispatching and data serialization library. ;-) - Jordan On Feb 28, 2014, at 10:51 AM, Kevin Bowling wrote: > Depending on what kind of development you are doing, either of these make a decent foundation library: > > * http://facebook.github.io/libphenom/ > * https://github.com/joyent/libuv