[ejabberd] Ejabberd hardware requirements for very high load

David Chan chan_hok_ching at yahoo.com
Fri Jan 13 19:31:14 MSK 2012


Hi Michael, Kirk:

Yeah I think 2 mil connections on a single machine is impressive indeed. 
But I think using a big fat machine alone (i.e. scaling UP) won't achieve that, they might have some special resource optimization software?
What I want to achieve in my project is like having 10+ (cheap) machines to handle large scale traffic. 
It will be great if configuring ejabberd (clustering) alone is enough to achieve this, without using some kind of rocket-science tools.

I came across Lobo loco's article before, I'll try on that some time later (I thought the clustering was easy... anyway), thx man!

David

________________________________
 From: "ejabberd-request at jabber.ru" <ejabberd-request at jabber.ru>
To: ejabberd at jabber.ru 
Sent: Thursday, 12 January 2012, 17:00
Subject: ejabberd Digest, Vol 102, Issue 9
 
Send ejabberd mailing list submissions to
    ejabberd at jabber.ru

To subscribe or unsubscribe via the World Wide Web, visit
    http://lists.jabber.ru/mailman/listinfo/ejabberd
or, via email, send a message with subject or body 'help' to
    ejabberd-request at jabber.ru

You can reach the person managing the list at
    ejabberd-owner at jabber.ru

When replying, please edit your Subject line so it is more specific
than "Re: Contents of ejabberd digest..."


Today's Topics:

   1. Re: Ejabberd hardware requirements for very high load
      (Michael Weibel)
   2. OAuth2.0 with SASL GSS-API: Support planned? (Michael Weibel)
   3. Re: Shared Roster / Groups with External    Authentication:
      Possible? (???)


----------------------------------------------------------------------

Message: 1
Date: Wed, 11 Jan 2012 12:50:32 +0100
From: Michael Weibel <michael.weibel+xmpp at gmail.com>
To: ejabberd at jabber.ru
Subject: Re: [ejabberd] Ejabberd hardware requirements for very high
    load
Message-ID:
    <CAFRrLeUtMTTG625k-mgts3EBEAw5+2VTBjw4vuacU_PQO5p9rA at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi David & Kirk,

>
> I'm actually presuming that they are still using ejabberd, yes partly
> because of the erlang bit in their articles, but also because they were
> previously looking for ejabberd experienced developers and I THINK they
> might have mentioned it a few years ago (when they started).
>
> It would be nice to hear a bit more detail about their setup, although I
> suspect that might count as company secrets :)
>

To hear about their setup would be absolutely nice and I hope someone of
them is reading this. :)


> Bear in mind that from the articles they are running those 2 million
> connections on a single machine.
>
> ..which is pretty impressive!


> Unfortunately one of the best guides to setting up ejabberd clustering,
> seems to have gone offline
> http://liferayz.com/how-to-setup-clustering-in-ejabberd-tutorial/ perhaps
> someone has an archived copy of that ? (its not on the wayback machine I
> checked).
>

I just setup 2 machines with ejabberd & clustered together. It's sad that
the documentation about it is pretty bad.. In the ejabberd guide it sounds
pretty easy but actually only half of the process is described there.
Using this link:
http://tdewolf.blogspot.com/2009/07/clustering-ejabberd-nodes-using-mnesia.html
And this conversation:
http://chatlogs.jabber.ru/ejabberd@conference.jabber.ru/2006/08/04.html

I was able to get the clustering working. It would be pretty nice if there
would be some more detailed documentation on the ejabberd website.

Regarding my original question: It's a little bit hard to guess what
whatsapp actually did to get to those 2 million connections, although they
mentioned their machine (which has a lot of cpu's and high memory). I think
they did additional tuning to reach that goal.

I'd be glad if anyone else could tell their experience regarding this.
But maybe I need to setup some small clustering system with a loadbalancer
and do a benchmark on top of it...
Does anyone know if this guide: http://www.ejabberd.im/benchmark is still
the way to go for benchmarking?

- Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jabber.ru/pipermail/ejabberd/attachments/20120111/a5233cb3/attachment-0001.html>

------------------------------

Message: 2
Date: Wed, 11 Jan 2012 17:15:57 +0100
From: Michael Weibel <michael.weibel+xmpp at gmail.com>
To: ejabberd <ejabberd at jabber.ru>
Subject: [ejabberd] OAuth2.0 with SASL GSS-API: Support planned?
Message-ID:
    <CAFRrLeVA-2yT4mK1_vEOsd4pto8AdqCWqz=m9T3Prr1muV1sog at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hey all,

does anyone know if there's support planned for the OAuth2.0 via SASL
GSS-API Specification[1]?
I know that ejabberd has added support for GSS-API since 2.1.9 but I'm not
really sure if this works without some additional development.

Thanks,
Michael

[1]: http://tools.ietf.org/html/draft-ietf-kitten-sasl-oauth-00
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jabber.ru/pipermail/ejabberd/attachments/20120111/e811d627/attachment-0001.html>

------------------------------

Message: 3
Date: Thu, 12 Jan 2012 13:15:12 +0800
From: ??? <langxianzhe at gmail.com>
To: ejabberd at jabber.ru
Subject: Re: [ejabberd] Shared Roster / Groups with External
    Authentication:    Possible?
Message-ID: <y3fx1i7lgp4gvbhaithjjje1.1326345312967 at email.android.com>
Content-Type: text/plain; charset=utf-8

I'm really interested to know how other developers have gone about integrating ejabberd with social websites.

i using  exmpp and nitrogen 

Tom Lancaster <tom at tnhvietnam.com>???

>Ok, still no luck adding users from external auth to a shared roster.
>So now I step back and ask the question:
>
>How do sites like mine deal with ejabberd auth when they want to
>mirror groups from the website in ejabberd to allow subsets of site
>users to converse with each other?
>
>I assumed that external authentication would be the way to go, but
>I've been down that alley for several weeks and it seems not to
>support the features I need.
>
>Do I need to manually add my site users to ejabberd on registration?
>What happens when they change their password?
>
>I'm really interested to know how other developers have gone about
>integrating ejabberd with social websites.
>
>Again, thank you for your time.
>
>Regards,
>
>Tom
>
>On Thu, Jan 5, 2012 at 8:15 PM, Tom Lancaster <tom at tnhvietnam.com> wrote:
>> Incidentally, here is some output from ejabberd.log around the time I
>> tried to connect to everybody at muc.chat.xemzi.com :
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.594.0>:ejabberd_c2s:1194) : broadcast
>> [{privacy_list,{userlist,"invisible",
>> ? ? ? ? ? ? ? ? ? ? ? ? [{listitem,jid,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?{"everybody","muc.chat.xemzi.com",[]},
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?allow,1,false,false,false,false,true},
>> ? ? ? ? ? ? ? ? ? ? ? ? ?{listitem,none,none,deny,2,false,false,false,false,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?true}],
>> ? ? ? ? ? ? ? ? ? ? ? ? false},
>> ? ? ? ? ? ? ? "invisible"}]
>>
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.594.0>:ejabberd_c2s:1400) : Send XML on stream = "<iq
>> from='growler at chat.xemzi.com' to='growler at chat.xemzi.com/Ostrich'
>> id='push2561196467' type='set'><query xmlns='jabber:iq:privacy'><list
>> name='invisible'/></query></iq>"
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.594.0>:ejabberd_c2s:1400) : Send XML on stream = "<iq
>> from='growler at chat.xemzi.com' to='growler at chat.xemzi.com/Ostrich'
>> id='iChat_175B2077' type='result'/>"
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.593.0>:ejabberd_receiver:320) : Received XML on stream =
>> "<presence to=\"everybody at muc.chat.xemzi.com/growler\"><x
>> xmlns=\"http://jabber.org/protocol/muc\"><history
>> maxstanzas=\"25\"/></x>\n</presence>\n"
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.593.0>:shaper:61) : State: {maxrate,1000,17.51430112697342,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?1325768941245865}, Size=137
>> M=69.10516432874128, I=346.669
>>
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.594.0>:ejabberd_router:301) : route
>> ? ? ? ?from {jid,"growler","chat.xemzi.com","Ostrich","growler",
>> ? ? ? ? ? ? ? ? ?"chat.xemzi.com","Ostrich"}
>> ? ? ? ?to {jid,"everybody","muc.chat.xemzi.com","growler","everybody",
>> ? ? ? ? ? ? ? ?"muc.chat.xemzi.com","growler"}
>> ? ? ? ?packet {xmlelement,"presence",
>> ? ? ? ? ? ? ? ? ? [{"to","everybody at muc.chat.xemzi.com/growler"}],
>> ? ? ? ? ? ? ? ? ? [{xmlelement,"x",
>> ? ? ? ? ? ? ? ? ? ? ? ?[{"xmlns","http://jabber.org/protocol/muc"}],
>> ? ? ? ? ? ? ? ? ? ? ? ?[{xmlelement,"history",[{"maxstanzas","25"}],[]}]},
>> ? ? ? ? ? ? ? ? ? ?{xmlcdata,<<"\n">>}]}
>>
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.594.0>:ejabberd_local:286) : local route
>> ? ? ? ?from {jid,"growler","chat.xemzi.com","Ostrich","growler",
>> ? ? ? ? ? ? ? ? ?"chat.xemzi.com","Ostrich"}
>> ? ? ? ?to {jid,"everybody","muc.chat.xemzi.com","growler","everybody",
>> ? ? ? ? ? ? ? ?"muc.chat.xemzi.com","growler"}
>> ? ? ? ?packet {xmlelement,"presence",
>>
>> ....
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.594.0>:ejabberd_sm:394) : session manager
>> ? ? ? ?from {jid,"growler","chat.xemzi.com","Ostrich","growler",
>> ? ? ? ? ? ? ? ? ?"chat.xemzi.com","Ostrich"}
>> ? ? ? ?to {jid,"everybody","muc.chat.xemzi.com","growler","everybody",
>> ? ? ? ? ? ? ? ?"muc.chat.xemzi.com","growler"}
>> ? ? ? ?packet {xmlelement,"presence",
>> ? ? ? ? ? ? ? ? ? ? ? ? ? [{"to","everybody at muc.chat.xemzi.com/growler"}],
>> ? ? ? ? ? ? ? ? ? ? ? ? ? [{xmlelement,[...],...},{xmlcdata,...}]}
>>
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.594.0>:ejabberd_sm:488) : packet droped
>>
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.593.0>:ejabberd_receiver:320) : Received XML on stream = "<iq
>> type=\"error\" to=\"growler at chat.xemzi.com\"
>> id=\"push2561196467\"><error type=\"501\">Cl
>> ient has no handler for IQ type set namespace
>> jabber:iq:privacy</error>\n</iq>\n"
>>
>> =INFO REPORT==== 2012-01-05 20:09:01 ===
>> D(<0.593.0>:shaper:61) : State: {maxrate,1000,206.34798465641498,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?1325768941592541}, Size=163
>> M=90.87604429712994, I=1.316
>>
>>
>>
>> On Thu, Jan 5, 2012 at 8:08 PM, Tom Lancaster <tom at tnhvietnam.com> wrote:
>>> On Thu, Jan 5, 2012 at 7:32 PM, Badlop <badlop at gmail.com> wrote:
>>>> 2012/1/3 Tom Lancaster <tom at tnhvietnam.com>:
>>>>> I have been wracking my brain and reading everything I can to
>>>>> establish whether I can accomplish the following using ejabberd:
>>>>>
>>>>> 1) Everyone in a shared roster (ie: all users of my website can see
>>>>> each other through an xmpp client)
>>>>
>>>> See:
>>>>
>>>> http://www.ejabberd.im/shared-roster-all#example1
>>>
>>>
>>> From that page: "This feature requires internal authentication. If you
>>> use external authentication, LDAP... then adding @all@ to a shared
>>> roster group will do nothing."
>>>
>>> Which is why I'm asking if there is any way at all to do shared roster
>>> groups with external authentication. In your previous email you said
>>> that @all@ would in fact work, in contradiction of the above linked
>>> page. I've followed those instructions and I'm not getting any kind of
>>> shared roster on iChat or Adium. This is sort of why I'm asking what I
>>> should expect to see on the client end if it was working: is there
>>> anything special I need to do to see the shared contacts from the
>>> server? I'm not seeing anyone show up in my buddy list.
>>>
>>> Thanks again for your time, and please excuse my confusion - I think I
>>> may be incorrectly conflating "group" and "group chat" which is why
>>> I'm asking if I need to make my client join a "group chat" in order to
>>> see the shared contacts.
>>>>
>>>>
>>>> ---
>>>> Badlop
>>>> ProcessOne
>>>> _______________________________________________
>>>> ejabberd mailing list
>>>> ejabberd at jabber.ru
>>>> http://lists.jabber.ru/mailman/listinfo/ejabberd
>_______________________________________________
>ejabberd mailing list
>ejabberd at jabber.ru
>http://lists.jabber.ru/mailman/listinfo/ejabberd

------------------------------

_______________________________________________
ejabberd mailing list
ejabberd at jabber.ru
http://lists.jabber.ru/mailman/listinfo/ejabberd


End of ejabberd Digest, Vol 102, Issue 9
****************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jabber.ru/pipermail/ejabberd/attachments/20120113/699add52/attachment-0001.html>


More information about the ejabberd mailing list