[ejabberd] Beam 100% CPU usage

Emmanuel Lacour elacour at easter-eggs.com
Sat Jan 9 02:03:17 MSK 2010


On Fri, Jan 08, 2010 at 03:42:16PM -0500, Brian Cully wrote:
> 	Try using "i()." in the debug console. Typically if something is
> 	burning CPU like that there will be a large mailbox associated with
> 	the offending process.
> 

here is the output ... I don't understand it and I don't understand what
"i()." does :(

do you see something weird?


Pid                   Initial Call                          Heap     Reds Msgs
Registered            Current Function                     Stack              
<0.0.0>               otp_ring0:start/2                     1597 22097659    0
init                  init:loop/1                              2              
<0.2.0>               erlang:apply/2                        6765   342089    0
erl_prim_loader       erl_prim_loader:loop/3                   5              
<0.4.0>               gen_event:init_it/6                   2584 22059446    0
error_logger          gen_event:fetch_msg/5                   11              
<0.5.0>               erlang:apply/2                        2584    20888    0
application_controlle gen_server:loop/6                        7              
<0.7.0>               application_master:init/4              233       47    0
                      application_master:main_loop/2          12              
<0.8.0>               application_master:start_it/4          233       91    0
                      application_master:loop_it/4             5              
<0.9.0>               supervisor:kernel/1                    987     2059    0
kernel_sup            gen_server:loop/6                       12              
<0.10.0>              rpc:init/1                             987   521536    0
rex                   gen_server:loop/6                       12              
<0.11.0>              global:init/1                         2584  1824114    0
global_name_server    gen_server:loop/6                       12              
<0.12.0>              erlang:apply/2                        2584  1602784    0
                      global:loop_the_locker/1                 4              
<0.13.0>              erlang:apply/2                         233        4    0
                      global:collect_deletions/2               6              
<0.14.0>              erlang:apply/2                         233        3    0
                      global:loop_the_registrar/0              2              
<0.15.0>              inet_db:init/1                        1597 17014081    0
inet_db               gen_server:loop/6                       12              
<0.17.0>              supervisor:erl_distribution/1          377      323    0
net_sup               gen_server:loop/6                       12              
<0.18.0>              erl_epmd:init/1                        377      147    0
erl_epmd              gen_server:loop/6                       12              
<0.19.0>              auth:init/1                            987    81885    0
auth                  gen_server:loop/6                       12              
<0.20.0>              net_kernel:init/1                     2584  1831300    0
net_kernel            gen_server:loop/6                       12              
<0.21.0>              inet_tcp_dist:accept_loop/2            610  1950858    0
                      prim_inet:accept0/2                      9              
<0.22.0>              net_kernel:ticker/2                    233   881529    0
                      net_kernel:ticker_loop/2                 3              
<0.23.0>              global_group:init/1                    233   182247    0
global_group          gen_server:loop/6                       12              
<0.24.0>              file_server:init/1                   10946 11157151    0
file_server_2         gen_server:loop/6                       12              
<0.25.0>              erlang:apply/2                        6765   924575    0
code_server           code_server:loop/1                       3              
<0.26.0>              supervisor_bridge:user_sup/1           987       86    0
                      gen_server:loop/6                       12              
<0.27.0>              user:server/2                         6765 32804989    0
user                  user:server_loop/2                       4              
<0.28.0>              kernel_config:init/1                   233       48    0
                      gen_server:loop/6                       12              
<0.29.0>              supervisor:kernel/1                    610      585    0
kernel_safe_sup       gen_server:loop/6                       12              
<0.39.0>              application_master:init/4              377       25    0
                      application_master:main_loop/2          12              
<0.40.0>              application_master:start_it/4        28657  1513459    0
                      application_master:loop_it/4             5              
<0.45.0>              application_master:init/4              233       25    0
                      application_master:main_loop/2          12              
<0.46.0>              application_master:start_it/4          233       90    0
                      application_master:loop_it/4             5              
<0.48.0>              supervisor:sasl/1                      233      194    0
sasl_sup              gen_server:loop/6                       12              
<0.49.0>              supervisor:sasl/1                      233      192    0
sasl_safe_sup         gen_server:loop/6                       12              
<0.50.0>              gen_event:init_it/6                    233       34    0
alarm_handler         gen_event:fetch_msg/5                   11              
<0.51.0>              overload:init/1                        233       45    0
overload              gen_server:loop/6                       12              
<0.52.0>              release_handler:init/1                 987     1341    0
release_handler       gen_server:loop/6                       12              
<0.54.0>              randoms:init/0                         233  3708202    0
random_generator      randoms:loop/0                           2              
<0.56.0>              supervisor:dets_sup/1                  377      646    0
dets_sup              gen_server:loop/6                       12              
<0.57.0>              dets_server:init/1                     377      989    0
dets                  gen_server:loop/6                       12              
<0.62.0>              application_master:init/4              233       25    0
                      application_master:main_loop/2          12              
<0.63.0>              application_master:start_it/4          233       48    0
                      application_master:loop_it/4             5              
<0.64.0>              supervisor:mnesia_sup/1                377      231    0
mnesia_sup            gen_server:loop/6                       12              
<0.65.0>              gen_event:init_it/6                    233     1338    0
mnesia_event          gen_event:fetch_msg/5                   11              
<0.66.0>              supervisor:mnesia_kernel_sup/1         377      677    0
mnesia_kernel_sup     gen_server:loop/6                       12              
<0.67.0>              mnesia_monitor:init/1                  233  2431098    0
mnesia_monitor        gen_server:loop/6                       12              
<0.68.0>              mnesia_subscr:init/1                   233      168    0
mnesia_subscr         gen_server:loop/6                       12              
<0.69.0>              mnesia_sp:init_proc/4                 1597 10306556    0
mnesia_locker         mnesia_locker:loop/1                    19              
<0.70.0>              mnesia_recover:init/1                 1597  3531876    0
mnesia_recover        gen_server:loop/6                       12              
<0.71.0>              mnesia_sp:init_proc/4                 6765 37661231    0
mnesia_tm             mnesia_tm:doit_loop/1                   20              
<0.75.0>              supervisor:disk_log_sup/1             1597  2149824    0
disk_log_sup          gen_server:loop/6                       12              
<0.76.0>              disk_log_server:init/1                2584  2498937    0
disk_log_server       gen_server:loop/6                       12              
(c)ontinue (q)uit -->
<0.95.0>              disk_log:init/2                       2584 27295856    0
                      disk_log:loop/1                          7              
<0.97.0>              timer:init/1                           610 20089288    0
timer_server          gen_server:loop/6                       12              
<0.98.0>              supervisor:mnesia_checkpoint_sup/      233       67    0
mnesia_checkpoint_sup gen_server:loop/6                       12              
<0.99.0>              supervisor:mnesia_snmp_sup/1           233       67    0
mnesia_snmp_sup       gen_server:loop/6                       12              
<0.100.0>             mnesia_controller:init/1               377   457063    0
mnesia_controller     gen_server:loop/6                       12              
<0.101.0>             mnesia_sp:init_proc/4                 2584      997    0
mnesia_late_loader    mnesia_late_loader:loop/1               15              
<0.125.0>             dets:init/2                            987     3044    0
                      dets:open_file_loop2/2                  12              
<0.161.0>             dets:init/2                            987 63174283    0
                      dets:open_file_loop2/2                  12              
<0.170.0>             dets:init/2                           4181   436961    0
                      dets:open_file_loop2/2                  12              
<0.175.0>             dets:init/2                            987 36327724    0
                      dets:open_file_loop2/2                  12              
<0.206.0>             dets:init/2                            987     3070    0
                      dets:open_file_loop2/2                  12              
<0.212.0>             application_master:init/4              233       25    0
                      application_master:main_loop/2          12              
<0.213.0>             application_master:start_it/4          233       62    0
                      application_master:loop_it/4             5              
<0.214.0>             supervisor:crypto_sup/1                233      124    0
crypto_sup            gen_server:loop/6                       12              
<0.215.0>             crypto_server:init/1                   610      406    0
crypto_server         gen_server:loop/6                       12              
<0.216.0>             supervisor:stringprep_sup/1            233      128    0
stringprep_sup        gen_server:loop/6                       12              
<0.217.0>             stringprep:init/1                      377      128    0
stringprep            gen_server:loop/6                       12              
<0.218.0>             ejabberd_app:init/0                    377      238    0
ejabberd              ejabberd_app:loop/1                      2              
<0.251.0>             supervisor:ejabberd_sup/1             2584     2014    0
ejabberd_sup          gen_server:loop/6                       12              
<0.252.0>             ejabberd_hooks:init/1                 2584     2207    0
ejabberd_hooks        gen_server:loop/6                       12              
<0.253.0>             ejabberd_node_groups:init/1            987      394    0
ejabberd_node_groups  gen_server:loop/6                       12              
<0.254.0>             pg2:init/1                            2584  1608241    0
pg2                   gen_server:loop/6                       12              
<0.255.0>             ejabberd_system_monitor:init/1        1597 28133023    0
ejabberd_system_monit gen_server:loop/6                       12              
<0.256.0>             ejabberd_router:init/1              832040 61052281    0
ejabberd_router       gen_server:loop/6                       12              
<0.259.0>             ejabberd_sm:init/1                     233      453    0
ejabberd_sm           gen_server:loop/6                       12              
<0.264.0>             ejabberd_s2s:init/1                    233      307    0
ejabberd_s2s          gen_server:loop/6                       12              
<0.267.0>             ejabberd_local:init/1                  610    51790    0
ejabberd_local        gen_server:loop/6                       12              
<0.270.0>             supervisor:ejabberd_tmp_sup/1         1597  6191312    0
ejabberd_receiver_sup gen_server:loop/6                       12              
<0.271.0>             supervisor:ejabberd_tmp_sup/1         4181   265655    0
ejabberd_c2s_sup      gen_server:loop/6                       12              
<0.272.0>             supervisor:ejabberd_tmp_sup/1         4181  2437083    0
ejabberd_s2s_in_sup   gen_server:loop/6                       12              
<0.273.0>             supervisor:ejabberd_tmp_sup/1         6765  4347872    0
ejabberd_s2s_out_sup  gen_server:loop/6                       12              
<0.274.0>             supervisor:ejabberd_tmp_sup/1          233       60    0
ejabberd_service_sup  gen_server:loop/6                       12              
<0.275.0>             supervisor:ejabberd_tmp_sup/1         2584    13247    0
ejabberd_http_sup     gen_server:loop/6                       12              
<0.276.0>             supervisor:ejabberd_tmp_sup/1          233       60    0
ejabberd_http_poll_su gen_server:loop/6                       12              
<0.277.0>             supervisor:ejabberd_tmp_sup/1          987     1237    0
ejabberd_iq_sup       gen_server:loop/6                       12              
<0.278.0>             supervisor:ejabberd_tmp_sup/1          233       60    0
ejabberd_frontend_soc gen_server:loop/6                       12              
<0.279.0>             supervisor:ejabberd_listener/1         610     1305    0
ejabberd_listeners    gen_server:loop/6                       12              
<0.280.0>             ejabberd_listener:init/3              1597    20408    0
                      prim_inet:accept0/2                     16              
<0.281.0>             ejabberd_listener:init/3              2584   992473    0
                      prim_inet:accept0/2                     16              
<0.283.0>             ejabberd_listener:init/3               610    17987    0
                      prim_inet:accept0/2                     16              
<0.284.0>             ejabberd_auth_ldap:init/1             2584  1374888    0
'ejabberd_auth_ldap_i gen_server:loop/6                       12              
<0.285.0>             eldap:init/1                        832040 42599569    0
'eldap_#Ref<0.0.0.892 gen_fsm:loop/7                          13              
<0.286.0>             supervisor_bridge:inet_gethost_na      233       51    0
inet_gethost_native_s gen_server:loop/6                       12              
<0.287.0>             inet_gethost_native:server_init/2     2584  1761715    0
inet_gethost_native   inet_gethost_native:main_loop/1          2              
<0.288.0>             eldap:init/1                          4181   496147    0
'eldap_#Ref<0.0.0.894 gen_fsm:loop/7                          13              
<0.289.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.290.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.293.0>             mod_announce:init/0                    233        7    0
'ejabberd_announce_im mod_announce:loop/0                      6              
<0.294.0>             mod_caps:init/1                     832040 42882162    0
'ejabberd_mod_caps_im gen_server:loop/6                       12              
<0.303.0>             gen_iq_handler:init/1                 6765  6614952    0
                      gen_server:loop/6                       12              
(c)ontinue (q)uit -->
<0.304.0>             gen_iq_handler:init/1                10946  4618465    0
                      gen_server:loop/6                       12              
<0.305.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.306.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.307.0>             supervisor:ejabberd_tmp_sup/1          233       60    0
'ejabberd_mod_irc_sup gen_server:loop/6                       12              
<0.308.0>             mod_irc:init/1                        2584   486846    0
'ejabberd_mod_irc_im. gen_server:loop/6                       12              
<0.309.0>             asn1rt_driver_handler:init/1           377      150    0
asn1_driver_owner     asn1rt_driver_handler:loop/0             1              
<0.311.0>             iconv:init/1                           377      130    0
iconv                 gen_server:loop/6                       12              
<0.314.0>             gen_iq_handler:init/1                  987    25573    0
                      gen_server:loop/6                       12              
<0.315.0>             gen_iq_handler:init/1                 2584  2341656    0
                      gen_server:loop/6                       12              
<0.316.0>             supervisor:ejabberd_tmp_sup/1         1597      311    0
'ejabberd_mod_muc_sup gen_server:loop/6                       12              
<0.317.0>             mod_muc:init/1                      121393   525213    0
'ejabberd_mod_muc_im. gen_server:loop/6                       12              
<0.323.0>             mod_muc_room:init/1                   4181     4468    0
                      gen_fsm:loop/7                          13              
<0.324.0>             mod_muc_room:init/1                   6765  1415253    0
                      gen_fsm:loop/7                          13              
<0.326.0>             mod_offline:init/1                    6765  4622294    0
'ejabberd_offline_im. mod_offline:loop/1                       3              
<0.328.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.330.0>             gen_iq_handler:init/1                 4181   561834    0
                      gen_server:loop/6                       12              
<0.331.0>             supervisor:mod_proxy65/1               610      285    0
'ejabberd_mod_proxy65 gen_server:loop/6                       12              
<0.332.0>             mod_proxy65_sm:init/1                  233       44    0
'ejabberd_mod_proxy65 gen_server:loop/6                       12              
<0.335.0>             supervisor:ejabberd_tmp_sup/1          233       60    0
'ejabberd_mod_proxy65 gen_server:loop/6                       12              
<0.336.0>             mod_proxy65_service:init/1            2584   876605    0
'ejabberd_mod_proxy65 gen_server:loop/6                       12              
<0.337.0>             ejabberd_listener:init/3               377      327    0
                      prim_inet:accept0/2                     16              
<0.338.0>             mod_pubsub:init/1                   196418 64702682    0
'ejabberd_mod_pubsub_ gen_server:loop/6                       12              
<0.339.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.340.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.341.0>             gen_iq_handler:init/1               514229 12696159    0
                      gen_server:loop/6                       12              
<0.342.0>             gen_iq_handler:init/1                 4181   310252    0
                      gen_server:loop/6                       12              
<0.350.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.351.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.356.0>             gen_iq_handler:init/1               121393  6208933    0
                      gen_server:loop/6                       12              
<0.373.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.374.0>             gen_iq_handler:init/1                  233       29    0
                      gen_server:loop/6                       12              
<0.375.0>             mod_vcard_ldap:init/1                 1597   278669    0
'ejabberd_mod_vcard_l gen_server:loop/6                       12              
<0.376.0>             eldap:init/1                       1346269 13411139    0
'eldap_#Ref<0.0.0.926 gen_fsm:loop/7                          13              
<0.377.0>             gen_iq_handler:init/1                 1597     8273    0
                      gen_server:loop/6                       12              
<0.855.0>             ejabberd_receiver:init/1              6765  2536997    0
                      gen_server:loop/6                       12              
<0.858.0>             ejabberd_c2s:init/1                   3595 18497739    0
                      erlang:hibernate/3                       0              
<0.1118.1056>         ejabberd_listener:init/3               987   464288    0
                      prim_inet:accept0/2                     16              
<0.18719.1104>        ejabberd_receiver:init/1               359    37744    0
                      erlang:hibernate/3                       0              
<0.18726.1104>        ejabberd_c2s:init/1                   3017   634595    0
                      erlang:hibernate/3                       0              
<0.21622.864>         ejabberd_receiver:init/1              1597   546188    0
                      gen_server:loop/6                       12              
<0.21623.864>         ejabberd_c2s:init/1                  75025 10465969    0
                      gen_fsm:loop/7                          13              
<0.20596.953>         ejabberd_receiver:init/1                68       71    0
                      erlang:hibernate/3                       0              
<0.1.954>             ejabberd_receiver:init/1                64       91    0
                      erlang:hibernate/3                       0              
<0.2214.954>          ejabberd_receiver:init/1                64       91    0
                      erlang:hibernate/3                       0              
<0.3476.954>          ejabberd_receiver:init/1                68       79    0
                      erlang:hibernate/3                       0              
<0.4337.1114>         ejabberd_receiver:init/1               610    23941    0
                      gen_server:loop/6                       12              
<0.4344.1114>         ejabberd_c2s:init/1                  17711   440600    0
                      gen_fsm:loop/7                          13              
<0.4436.954>          ejabberd_receiver:init/1                68       79    0
                      erlang:hibernate/3                       0              
<0.6946.954>          ejabberd_receiver:init/1                64       99    0
                      erlang:hibernate/3                       0              
<0.8074.954>          ejabberd_receiver:init/1                68       71    0
                      erlang:hibernate/3                       0              
(c)ontinue (q)uit -->
<0.18866.1059>        ejabberd_receiver:init/1              2584   129347    0
                      gen_server:loop/6                       12              
<0.18867.1059>        ejabberd_c2s:init/1                   6765  1968003    0
                      gen_fsm:loop/7                          13              
<0.20066.1059>        ejabberd_receiver:init/1              2584  1399625    0
                      gen_server:loop/6                       12              
<0.20067.1059>        ejabberd_s2s_in:init/1                6765  3967830    0
                      gen_fsm:loop/7                          13              
<0.26913.700>         ejabberd_receiver:init/1               610   905217    0
                      gen_server:loop/6                       12              
<0.26957.700>         ejabberd_c2s:init/1                  75025 15137690    0
                      gen_fsm:loop/7                          13              
<0.31201.1108>        ejabberd_receiver:init/1              4181    51393    0
                      gen_server:loop/6                       12              
<0.31202.1108>        ejabberd_c2s:init/1                   2579   397190    0
                      erlang:hibernate/3                       0              
<0.31284.1108>        ejabberd_receiver:init/1              4181    26839    0
                      gen_server:loop/6                       12              
<0.31285.1108>        ejabberd_c2s:init/1                   2175   207317    0
                      erlang:hibernate/3                       0              
<0.3251.1117>         ejabberd_receiver:init/1              1597    17772    0
                      gen_server:loop/6                       12              
<0.3252.1117>         ejabberd_c2s:init/1                  28657   341097    0
                      gen_fsm:loop/7                          13              
<0.3648.1117>         ejabberd_receiver:init/1               391     2504    0
                      erlang:hibernate/3                       0              
<0.3649.1117>         ejabberd_s2s_in:init/1                2584     4390    0
                      gen_fsm:loop/7                          13              
<0.3650.1117>         gen:init_it/6                         2584     6121    0
                      p1_fsm:process_message/8                14              
<0.3651.1117>         ejabberd_receiver:init/1               354      962    0
                      erlang:hibernate/3                       0              
<0.3669.1117>         gen:init_it/6                         2584     2772    0
                      p1_fsm:process_message/8                14              
<0.3670.1117>         gen:init_it/6                         6765     7908    0
                      p1_fsm:process_message/8                14              
<0.3671.1117>         gen:init_it/6                         4181     6213    0
                      p1_fsm:process_message/8                14              
<0.3672.1117>         gen:init_it/6                         1597     6118    0
                      p1_fsm:process_message/8                14              
<0.3673.1117>         gen:init_it/6                         2584     2148    0
                      p1_fsm:process_message/8                14              
<0.3674.1117>         ejabberd_receiver:init/1               337      423    0
                      erlang:hibernate/3                       0              
<0.3675.1117>         ejabberd_receiver:init/1               337      427    0
                      erlang:hibernate/3                       0              
<0.3676.1117>         ejabberd_receiver:init/1               352      925    0
                      erlang:hibernate/3                       0              
<0.3677.1117>         ejabberd_receiver:init/1               354      968    0
                      erlang:hibernate/3                       0              
<0.3678.1117>         ejabberd_receiver:init/1               380     1724    0
                      erlang:hibernate/3                       0              
<0.3679.1117>         ejabberd_s2s_in:init/1                4181    11204    0
                      gen_fsm:loop/7                          13              
<0.3682.1117>         ejabberd_receiver:init/1               434      296    0
                      erlang:hibernate/3                       0              
<0.3721.1117>         ejabberd_receiver:init/1               391      955    0
                      erlang:hibernate/3                       0              
<0.3722.1117>         ejabberd_s2s_in:init/1                4181     2524    0
                      gen_fsm:loop/7                          13              
<0.3723.1117>         gen:init_it/6                         2584     3941    0
                      p1_fsm:process_message/8                14              
<0.3724.1117>         ejabberd_receiver:init/1               352     1013    0
                      erlang:hibernate/3                       0              
<0.3727.1117>         ejabberd_receiver:init/1               380     3192    0
                      erlang:hibernate/3                       0              
<0.3728.1117>         ejabberd_s2s_in:init/1                6765    10860    0
                      gen_fsm:loop/7                          13              
<0.3776.1117>         ejabberd_receiver:init/1              4181     1110    0
                      gen_server:loop/6                       12              
<0.3777.1117>         ejabberd_s2s_in:init/1                2584     3509    0
                      gen_fsm:loop/7                          13              
<0.3778.1117>         inet_tcp_dist:do_accept/6              610      486    0
                      dist_util:con_loop/9                    11              
<0.3782.1117>         erlang:apply/2                       10946     2839    0
                      shell:shell_rep/4                       17              
<0.3783.1117>         erlang:apply/2                        1597    86505    0
                      rpc:local_call/3                        51              
<0.27233.685>         ejabberd_receiver:init/1              1597   536870    0
                      gen_server:loop/6                       12              
<0.27264.685>         ejabberd_c2s:init/1                   2205  7616311    0
                      erlang:hibernate/3                       0              
<0.19481.974>         erlang:apply/2                        4181     4994    0
                      file_io_server:server_loop/1             3              
<0.22996.1110>        ejabberd_receiver:init/1               610    58110    0
                      gen_server:loop/6                       12              
<0.22997.1110>        ejabberd_c2s:init/1                   2845  3775638    0
                      erlang:hibernate/3                       0              
Total                                                    5350021 33494110    0
                                                            1956              
ok


More information about the ejabberd mailing list