Crash in ostream <<operator

[Willem Jan Withagen]

On 15-3-2016 19:52, Dimitry Andric wrote:
> On 15 Mar 2016, at 12:41, Willem Jan Withagen <wjw at digiware.nl> wrote:
>>
>> While running Ceph tools I get a crash in
>> fr 10
>> #10 0x00000000016d82ca in FileStore::omap_get_values(coll_t const&, ghobject_t const&, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, ceph::buffer::list, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, ceph::buffer::list> > >*) ()
>> (gdb) l
>> 95      int preload_erasure_code()
>> 96      {
>> 97        string plugins = g_conf->osd_erasure_code_plugins;
>> 98        stringstream ss;
>> 99        int r = ErasureCodePluginRegistry::instance().preload(
>> 100         plugins,
>> 101         g_conf->erasure_code_dir,
>> 102         &ss);
>> 103       if (r)
>> 104         derr << ss.str() << dendl;
>> (gdb)
>> 105       else
>> 106         dout(10) << ss.str() << dendl;
>> 107       return r;
>> 108     }
>> 109
>>
>> All of this seems to be inlined since I'm not able to get at ss or r
>>
>>
>> #8  0x0000000000e16145 in std::__1::char_traits<char>::length (__s=0x0) at /usr/include/c++/v1/string:640
>> 640         static inline size_t length(const char_type* __s) {return strlen(__s);}
> 
> What happened here is that something attempted to initialize a
> std::string with a NULL pointer, and that isn't allowed.  As you saw in
> the debugger, the constructor just runs strlen() on the incoming string,
> and that will segfault.
> 
> 
>> Looking at the strlen implementation in
>>    /usr/srcs/head/src/lib/libc/string/strlen.c
>>
>> shows that strlen does not take 0x0 as pointer, so when we get here with __s = 0x0 all is lost.
>> So I tried running it through 3.7, but since this is in the libraries with the bintools/os, I'd expect
>> both versions to crash on this.
>>
>> Now the question I have to solve:
>> 	is it the compiler/toolset/libraries
>> 	is it a bug in the ceph code.
> 
> Most likely a bug in the Ceph code.  Try figuring out where the NULL
> pointer originally came from.

I've started with compiling wit -O0 but that probably will still inline
the pieces of code designated as such. Otherwise I'll have to resort to
inserting asserts.

I'm now using gdb 7.1, loading lldb gives me a sort of strange
commandline that looks like utf-8-ish???
Would lldb give better analysis of the structures?

--WjW