libhci update

[Bruce Simpson]

Hi guys,

I really hate to be long-winded with this... but it seems there are a 
number of thorny issues
in the Bluetooth area which we need to carefully consider and act upon 
going forward.

Iain Hibbert wrote:
> I looked briefly at BlueCove the other day and it seems to use a module to
> interface with the BlueZ/Linux API but it also has Windows and Mac modules
> amongst others. If it needs a FreeBSD or NetBSD module then that doesn't
> seem so difficult?
>   

I had literally spent two evenings before staring at 
PyBluez/LightBlue/BlueCove
code on my ThinkPad, whilst lying on my mother's sofa, and after hacking 
MLDv2
into FreeBSD-CURRENT's IPv6 stack.

When I hacked on libhci I was on a train from Glasgow to London last
Thursday evening, and let me tell you, I was staring at KScope a lot!

Of course it's also to do with the fact that BlueZ libhci is an API which is
being used by a number of apps out there, so I had more example material 
to work
from, having waded into this knowing NOTHING about the existing APIs.

If you look at the differences between netbt, FreeBSD bluetooth, and BlueZ,
it's ifdef city. Nothing is all that different under the hood in terms 
of meaning,
or role within the stack, but every possible thing is named differently.

In terms of porting/compatibility, my focus has been on developing shims
for the BlueZ API, given its relative wide adoption in the open source
software available for Bluetooth, rather than hacking BSD native support 
code,
as this just seemed like the quicker path towards my goal: Python support
for building Bluetooth server apps, on a FreeBSD based system.

I also wanted to minimize the diff sets required for interoperability. In my
experience, the smaller patches are, the more likely third parties are to
take them on upstream.


> I looked at LightBlue but I don't remember anything about it, was it
> perchance really incomplete and very BlueZ/Linux based?
>   

Yes, LightBlue is actually quite incomplete, although it is intended to 
be dog simple.

Some technical detail follows:

LightBlue relies on PyBlueZ as a prerequisite, and uses a handful of 
libhci functions, as well
as requiring OpenOBEX. It is also GPLv3'd which is off-putting to say 
the least
for commercial use.

Trouble is, PyBlueZ is actually very tied to some BlueZ specifics, i.e. 
the HCI interface
in particular. Any time we need to reach down to baseband, which you end 
up doing to
implement neighbor discovery, you need to use HCI. SDP alone just 
doesn't cut it;
paging every device in the immediate vicinity is quite an expensive 
operation.

It is the only Python native binding I know of which ties OBEX on top of 
Bluetooth
in a similar way to that of Java JSR-82's 'SessionNotifier' interface -- 
which is what we
really need for building Bluetooth server apps. We found Python's 
HTTPServer very
useful indeed, and having something like an OBEXServer would be just 
right for us.

However, LightBlue has the limitation that it doesn't have any support 
for multiple
devices/enumeration, unless you go directly to PyBlueZ, and doesn't 
support a
event driven interface for Bluetooth server apps -- its APIs are 
strictly synchronous
function calls.
Same for its OBEXClient class, however at least that can have some 
methods overridden,
but still isn't as flexible as we would like it to be.

To be fair, it isn't a lot of code, and implementing a new Python API 
similar to
LightBlue would not take too much time, provided the foundations are in 
place.

The real fly in the soup is the fact that the BlueZ bindings I've seen, 
are all tied
to BlueZ's use of a 16 bit wide identifier for each local Bluetooth 
device in
device enumeration. These end up leaking into the APIs. That's just 
unavoidable,
as it is still reasonable that there is a means of referring to devices 
without
opening them. In an ideal world we'd be able to use a string identifier, but
that is not what's happened NOW.

JSR-82 is less tied up in this way, but it still assumes that there is 
an integer
device ID unique to each local device which will fit in a Java JNI jint 
type.

...

To summarize: I'm just concerned that in focusing too much on building a
clean / BSD native API, we risk sidelining ourselves. There has been a 
lot of Bluetooth
activity in the Linux community, and by not having at least some degree of
source level compatibility, we risk not being able to leverage and make use
of that work which we could otherwise benefit from for small change cost.

There is no 'standard' for high level Bluetooth APIs that I am aware of, 
other than
JSR-82, and wider adoption of a platform tends to wind up with it being 
'de-facto'
standard. Whether we like it or not, it's a harsh reality of software work.
Hence my comparison to Betamax/VHS in my original reply.

I myself, I just get sick and tired of having to HACK HACK HACK every time
someone writes a neato little app for BlueZ. This situation is endemic 
simply
because Linux has a popularity which the BSDs don't, as such it can 
potentially
attract a greater number of folk getting into software dev for the first 
time,
or on a hobbyist basis. I realize that this might be an unpopular view 
in the BSD
camp, but that is how it looks from where I'm standing.

[I've been living on the edge with this stuff for months, and I really 
just want
to make progress so I can focus on more fun things in life than fscking 
computers!!]

> so write a module that interfaces (for example) the Java (BlueCove?) API
> to the FreeBSD OS layer. Its not that different from the BlueZ/Linux API
> and you can probably just do some copy and paste work. Thats how the GPL
> world works. Yes, you will not be able to integrate that work directly
> into FreeBSD but then I doubt a Java interface is ever going to be
> accepted into base anyway. Donate it to BlueCove.
>   

Well, that's exactly my point, I am not for one moment suggesting we 
fold JSR-82
into any base system... but time is critical and we need to work with 
what's already there,
unless someone comes up with a compelling alternative NOW.

> If its commercial, get those companies to contribute some BSD code.
>   

That's a reasonable statement, but for the fact that...

...The companies either don't exist or are potential competitors who aren't
using BSD or Linux -- or, WE are the potential companies!

My partner and I are already working on this, so it is very much a matter of
beg borrow or steal. There seems no sense, to my mind, in reinventing 
wheels...

There are competing interests in this space, but they've had the CapEx to go
off and implement their own platforms. As to the business end of it, whether
that's a tax write-off for investment purposes I know and care not, it's 
just
what any new player would be up against in the Bluetooth space.

>   
>> Thanks for this. I would far rather not introduce a runtime or link-time
>> dependency on -lnetgraph if I can possibly avoid it. I'll digest further
>> and try to see if this can be incorporated.
>>     
>
> But I thought, on FreeBSD the whole bluetooth stack is netgraph based..?
>   

It is, but generally the libbluetooth / libsdp libraries 'get away' with 
not touching
Netgraph directly, as the Netgraph ng_btsocket node exposes a number of 
socket options
and ioctls which require no knowledge/linkage to Netgraph in order to use.

I just ended up going to Netgraph to perform BlueZ style device 
enumeration, which
needs an integer handle (a bit like an ifnet ifindex in the network 
stack), rather than
a textual name.

The code which Maksim posted would help us to side step this in any 
prospective
libhci compatibility library.

> My stance on the compatibility issue is that there are some things in the
> BlueZ/Linux C API (the major thing being 'devid' to address the radio)
> that are tied to the actual OS support and are unsupportable unless you
> provide exactly the same API in the OS. But, OS support is way too low
> level for an application to deal (with as you say), and a higher level API
> is needed that does not contain such specifics.
>   

Sure, but I don't have free time to come up with such a higher level 
API, and as such,
I need to work with what's already out there.

Of course, if anyone else is willing to volunteer to work on this, they 
are more than
welcome to do so, but the problem with assembling a product / strategy 
based on that,
is, we need to know how much it's going to cost, and how much time it's 
going to take,
i.e. when will it be ready.

Hence my interest in leveraging what is already out there *NOW*. It is 
in no way a
technical or political endorsement of any particular approach, camp, 
product, or
philosophy, it is a purely pragmatic approach to the reality of working with
limited resources and time. :-)

As I say, I'm not trying to tread on anyone's toes, or otherwise rule 
out good and
technically valid solutions. We just needed this thing months ago and 
it's not there.

> The BlueZ guys are, I think, working on a dbus API that will be used by
> GNOME and KDE and hopefully it won't be tied to the Linux OS API so
> closely, so that we can write dbus modules and have applications just work
> on our OS.  I have not been providing any input or review of that API
> though, it would be good if somebody would step up and point out where the
> API is tied too closely to the Linux OS interface and get them to make it
> a bit more generic.
>   

Yes they are, however, as I've hinted at before, they seem focused on 
this and little
else. Back in January my colleague Alexei (Cc'd)and I did some work with 
the latest
BlueZ bluetoothd frm SVN. We were astonished and dismayed to find that 
operations
critical to the receiving of files had completely changed, i.e. the 
'inquiry scan'
setting, and the documentation had not been updated.

Whilst we had discovered and found the problem, and documented a workaround,
we had a demo suffer from a technical hitch because of this issue.

Of course, it is fair to argue that we get what we deserve for trying to 
work with
the bleeding edge of development. However, this wasn't just a few minor 
changes,
the entire configuration mechanism had been rewritten, with no documentation
other than the code.

I could care less about Linux or BSD specifics at this point. If folk 
want to create
new APIs, please do so -- I ain't stopping you! And if I can help out, I 
will.

I'm certainly receptive to everyone's ideas here given their technical 
merit, but it
should be borne in mind that we have a very specific goal.


>   
>> I looked at the Bluetooth specs and I can see that the inquiry sequence
>> doesn't hog all of the radio spectrum in use, but the implementation on
>> the CSR dongles won't raise any other events whilst the inquiry is in
>> progress.
>>     
>
> Is this purely a CSR problem?  My laptop has a Broadcom chip in and I
> notice that it can make multiple connections concurrently in that on
> bootup, it connects to both my mouse and keyboard by itself sometimes -
> the CSR dongle I used previously would connect to the keyboard fine but
> always fail the second connect with "Command Disallowed".  So much so that
> I thought perhaps about serialising connection attempts in the kernel.
>   

I doubt that the issue with inquiry tying up the controller is limited 
to CSR dongles,
but it's one area where we either need to engage the vendor directly and 
ask them what
the story is, or conduct experiments to map out the behaviour in the 
time domain
amongst different vendors and models (very time consuming).

If you look at the specs, inquiry keeps the controller quite busy. There 
are several different
inquiry variants, and they all involve frequency hopping, and the 
transmission of the
inquiry sequence. It's mostly baseband stuff and doesn't involve e.g. 
L2CAP layer.

The inquiry sub-state, as described in the spec, doesn't preclude other 
HCI events
during the inquiry, however it does recommend that ACL transports be parked,
as it only reserves slots for SCO.

I haven't delved deep enough to see if FreeBSD Bluetooth is doing ACL 
park when
periodic inquiry is active, I wager it doesn't, and what we're actually 
seeing is the
client device attempting to reconnect after the inquiry sub-state is 
left, after
seeing no reply.


> I've never looked at periodic inquiry though..
>   

Periodic inquiry is basically the same as regular inquiry, with the 
exception that the
periodic timer is hosted by the microcontroller itself. When the timer 
fires, it will
drop just about everything else that it's doing, mind you I've seen with 
OBEX sessions
running at the same time, it will either pick them up once inquiry has 
finished, or
finish the OBEX session (RFCOMM) before starting inquiry.

Of course given that it will not generate any event upcall at HCI layer 
for any other
baseband protocol event during the inquiry, it's of very limited 
usefulness for real apps.

The only application I've seen which uses it is the Bluetooth 
scanner/sniffer/brute forcer
'Fine Tooth Comb' from shmoo.com.

> I have written at least a set of SDP primitives that I'm intending to
> import to NetBSD 'soon' (I have only one computer and am concentrating on
> 5.0 release first because running different OS versions is messy)
>
> I think the latest archive was at
>   http://www.netbsd.org/~plunky/sdp-20090227.tar.gz
>   

Thanks for this, I'll be sure to take a look once I can download it.

I don't seem to be able to reach www.netbsd.org at the moment, either 
from my home
ISP connection, or from freefall.freebsd.org. :-(

I am very grateful for your input and feedback, and hope that we can get 
this
ball rolling faster.

cheers,
BMS