[ejabberd] ejabberd and amnesia crashes

Fabio Ricci fabioricci75 at gmail.com
Tue Feb 9 15:58:29 MSK 2010


Hi,
We have a 4 node ejabber installation and we manage about 20 kPccu
In the last week we had 3 ejabber crash (on all the nodes)


The component that is crashing is mnesia.
This kind of crash has been introduced during last maintenance when we
disabled the mod_pubsub in ejabber.cfg
*%%  {mod_pubsub,   [ % requires mod_caps
%%              {access_createnode, pubsub_createnode},
%%              {plugins, ["default", "pep"]}
%%      ]},

*At each crash we have a mnesia coredump
the reason why is crashing is in the ejabberd.log file:
=ERROR REPORT==== 2010-02-04 21:44:23 ===
*Mnesia('ejabberd at jabbr001 <ejabberd at lol-jabbr001-prod.lls21.vda.goa>'): **
ERROR ** (core dumped to file:
"/data/ejabberd/bin/MnesiaCore.ejabberd at jabbr001_1265_316263_657059"</data/ejabberd/bin/MnesiaCore.ejabberd at lol-jabbr001-prod.lls21.vda.goa_1265_316263_657059>
)
 ** FATAL ** Cannot open log file
"/data/ejabberd/database/ejabberd at jabbr001/PREVIOUS.LOG"</data/ejabberd/database/ejabberd at lol-jabbr001-prod.lls21.vda.goa/PREVIOUS.LOG>:
{file_error,

"/data/ejabberd/database/ejabberd at jabbr001/PREVIOUS.LOG"</data/ejabberd/database/ejabberd at lol-jabbr001-prod.lls21.vda.goa/PREVIOUS.LOG>
,

system_limit}
*
We have noticed a lot of this messages from the last maintenance:

*=ERROR REPORT==== 2010-02-09 13:11:20 ===
Mnesia('ejabberd at jabbr001 <ejabberd at lol-jabbr001-prod.lls21.vda.goa>'): **
WARNING ** Mnesia is overloaded: {dump_log,

write_threshold}*

They are more frequent as the number of connection to the jabber increase.
During Low population hours we have one message every 30mins, during high
population we are flooded by this message. (also every second)

I've found interesting informations here:
http://streamhacker.com/2008/12/10/how-to-eliminate-mnesia-overload-events/
In my understaing we can't change that parameter directly in the ejabber
configuration file.

My idea is that we are reaching some system_limit internal to ejabber (there
are no limits on the host itself, memory cpu and i/o are ok)

Talking about limit we can also see errors like this just before the crash
happen:
*=ERROR REPORT==== 2010-02-08 20:38:12 ===
** Generic server <0.24022.18> terminating
** Last message in was {become_controller,<0.24023.18>}
** When Server state == {state,
                            {tlssock,#Port<0.4195053>,#Port<0.4196250>},
                            tls,none,undefined,65536,
                            {xml_stream_state,undefined,#Port<0.4208404>,[],
                                0,65536},
                            infinity}
** Reason for termination ==
** {system_limit,[{erlang,open_port,[{spawn,expat_erl},[binary]]},
                  {xml_stream,new,2},
                  {ejabberd_receiver,handle_call,3},
                  {gen_server,handle_msg,5},
                  {proc_lib,init_p,5}]}*

This may say that the limit is in the erlang open_port.
If we look at our configuration we have


*# ERL_MAX_PORTS: Maximum number of simultaneously open Erlang ports
#
# ejabberd consumes two or three ports for every connection, either
# from a client or from another Jabber server. So take this into
# account when setting this limit.
#
# Default: 32000
# Maximum: 268435456
#
ERL_MAX_PORTS=32000

*This is set to the minimum. I think that raising this parameter can be
worth a try.


Thanks for any suggestion, it will be much appreciated :)
Fabio

ejabberdctl.cfg:

POLL=true
SMP=auto
ERL_MAX_PORTS=32000
ERL_PROCESSES=2500000
ERL_MAX_ETS_TABLES=1400
ERLANG_NODE=ejabberd at jabber001

ejabberd.cfg:

{loglevel, 2}.
{hosts, ["ex.net"]}.
{listen,
 [
    {5222, ejabberd_c2s, [
        {certfile, "/data/ejabberd/conf/ssl.pem"}, starttls,
        {access, c2s},
        {shaper, c2s_shaper},
        {max_stanza_size, 65536}
    ]},
    {5223, ejabberd_c2s, [
        {certfile, "/data/ejabberd/conf/ssl.pem"}, tls,
        {access, c2s},
        {shaper, c2s_shaper},
        {max_stanza_size, 65536}
    ]},
    {5269, ejabberd_s2s_in, [
        {shaper, s2s_shaper},
        {max_stanza_size, 131072}
    ]},
    {5280, ejabberd_http, [
        http_bind,
        http_poll,
        web_admin
    ]}
 ]}.
{auth_method, [my, my_my]}.
{my_server_ingame,"http://1.2.8.4:8/authenticate"}.
{odbc_server, {mysql, "7.2.8.3", "ejabberd", "ejabberd", "password"}}.
{odbc_pool_size, 40}.
{odbc_keepalive_interval, 3600}.
{shaper, normal, {maxrate, 2000}}.
{shaper, fast, {maxrate, 50000}}.
{acl, admin, {user, "admin", "ex.net"}}.
{acl, local, {user_regexp, ""}}.
{access, max_user_sessions, [{4, all}]}.
{access, local, [{allow, local}]}.
{access, c2s, [{deny, blocked},
        {allow, all}]}.
{access, c2s_shaper, [{none, admin},
        {normal, all}]}.
{access, s2s_shaper, [{fast, all}]}.
{access, announce, [{allow, admin}]}.
{access, configure, [{allow, admin}]}.
{access, muc_admin, [{allow, admin}]}.
{access, muc, [{allow, all}]}.
{access, register, [{allow, all}]}.
{access, pubsub_createnode, [{allow, all}]}.
{language, "en"}.
{modules,
 [
  {mod_adhoc,    []},
  {mod_caps,     []},
  {mod_disco,    []},
  {mod_http_bind,[]},
  {mod_irc,      []},
  {mod_last_odbc,     []},
  {mod_muc,      [
                  {min_message_interval, 0.4},
                  {min_presence_interval, 4},
          {access, muc},
          {access_create, muc},
          {access_persistent, muc},
          {history_size, 0},
          {default_room_options,
           [
                {max_users, 1200},
            {anonymous, false},
            {public, false}
           ]},
          {access_admin, muc_admin}
         ]},
  {mod_offline_odbc,  []},
  {mod_privacy,  []},
  {mod_private_odbc,  []},
  {mod_register, [
        {welcome_message, {"Welcome!",
        "Welcome to this Jabber server."}},
        {access, register}
    ]},
  {mod_roster_odbc,   []},
  {mod_shared_roster,[]},
  {mod_time,     []},
  {mod_vcard_odbc,    []},
  {mod_xmlrpc,[{port,4560},{timeout,5000}]},
  {mod_version,  []}
 ]}.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jabber.ru/pipermail/ejabberd/attachments/20100209/02c4c5b0/attachment-0001.html>


More information about the ejabberd mailing list