[ejabberd] S2S overloaded

Carlos Abalde carlos.abalde at gmail.com
Tue Aug 25 20:17:19 MSD 2009


>> ejabberd_s2s_in processes
>> started queuing requests in their mailboxes (up to message_queue_len's
>> greater than 400000 stanzas in one of the nodes!).
> 
> Are you sure the problem is in s2s_in? What does process_info(Pid,
> current_function) show? Do you have a shaper enabled?

Yes. I'm sure the problems in s2s_in which is queuing requests. When
things started to work wrong I looked for the processes with the largest
queues in the overloaded cluster and got their info. For example, you
can see the output of process_info(pid(0,12721,0)) in the next lines (I
changes user ids for privacy):

[{current_function,{gen,wait_resp_mon,3}},
 {initial_call,{proc_lib,init_p,5}},
 {status,waiting},
 {message_queue_len,341737},
 {messages,[{'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","2 at j9.dom.com"}],

[{xmlelement,"status",[],[]},

{xmlelement,"priority",[],[{xmlcdata,<<"0">>}]}]}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","3 at j9.dom.com"},
                                                         {"type","probe"}],
                                                        []}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","3 at j9.dom.com"}],

[{xmlelement,"status",[],[]},

{xmlelement,"priority",[],[{xmlcdata,<<"0">>}]}]}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","4 at j9.dom.com"},
                                                         {"type","probe"}],
                                                        []}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","4 at j9.dom.com"}],

[{xmlelement,"status",[],[]},

{xmlelement,"priority",[],[{xmlcdata,<<...>>}]}]}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","5 at j9.dom.com"},
                                                         {"type","probe"}],
                                                        []}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","5 at j9.dom.com"}],

[{xmlelement,"status",[],[]},

{xmlelement,"priority",[],[{...}]}]}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","6 at j9.dom.com"},
                                                         {"type","probe"}],
                                                        []}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","1 at j17.dom.com/dom"},

{"to","6 at j9.dom.com"}],

[{xmlelement,"status",[],[]},

{xmlelement,"priority",[],...}]}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","7 at j17.dom.com/dom"},

{"to","59532223 at j9.dom.com"},
                                                         {"type",[...]}],
                                                        []}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from","7 at j17.dom.com/dom"},{"to",[...]}],

[{xmlelement,[...],...},{xmlelement,...}]}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",

[{"from",[...]},{[...],...},{...}],
                                                        []}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",
                                                        [{[...],...},{...}],
                                                        [{...}|...]}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",
                                                        [{...}|...],
                                                        []}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,"presence",
                                                        [...],...}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,[...],...}}},
            {'$gen_event',{xmlstreamelement,{xmlelement,...}}},
            {'$gen_event',{xmlstreamelement,{...}}},
            {'$gen_event',{xmlstreamelement,...}},
            {'$gen_event',{...}},
            {'$gen_event',...},
            {...}|...]},
 {links,[<0.224.0>]},
 {dictionary,[{'$ancestors',[ejabberd_s2s_in_sup,
                             ejabberd_sup,<0.46.0>]},
              {'$initial_call',{ejabberd_s2s_in,init,1}}]},
 {trap_exit,false},
 {error_handler,error_handler},
 {priority,normal},
 {group_leader,<0.45.0>},
 {total_heap_size,74732575},
 {heap_size,74732575},
 {stack_size,68},
 {reductions,42258651},
 {garbage_collection,[{fullsweep_after,0},{minor_gcs,0}]},
 {suspending,[]}]

----

Then, lots of received xmlstreamelement's are waiting for s2s_in to be
processed. I have a S2S shaper enabled, but with a large enough value (I
think). Anyway, I'm going to repeat the experiment without shaper.

Do you detect any hint of the problem in the previous output? Thank you
very much for your reply!

--
Carlos.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: OpenPGP digital signature
URL: <http://lists.jabber.ru/pipermail/ejabberd/attachments/20090825/adaa5931/attachment-0001.pgp>


More information about the ejabberd mailing list