[ejabberd] Number of Pubsub nodes.

Andy Skelton skeltoac at gmail.com
Mon May 18 19:38:55 MSD 2009


On Mon, May 18, 2009 at 5:21 AM, Paul Kinlan <paul.kinlan at gmail.com> wrote:
> I need to potentially create 10,000+ pubsub nodes - dynamically, I need to
> understand if there is an upper limit to the number of nodes I can create
> either configureable or system programmed limit (such as the number of nodes
> is limited to the size of a byte integer or not).  I would also be
> interested if anyone has handled such loads before and I would be interested
> in understanding what they noticed the limits of the system to be - for
> example, demands on the systems, unforseen limitations with the number of
> pubsub nodes on the system due to memory, processor power, disk space and
> bandwidth etc.

I can't give you any performance data because I haven't put many users
on the it, but I can tell you about my setup. Maybe you can get some
ideas on how to optimize.

I have several million pubsub nodes. Only a handful are unique and
require records in the pubsub_node table. The rest are functionally
identical so I made them virtual. When pubsub goes to look up a node,
it asks the nodetree. I created a nodetree plug-in that checks the
node path and either reads the pubsub_node table or creates a record
out of thin air. This makes node lookups fast for both types of nodes:
virtual nodes because there is no disk access, and actual nodes
because the pubsub_node table is kept small. It also means I don't
have to create the millions of virtual nodes. They exist whenever I
use them. The only table I have to scale up is pubsub_state, and I
plan to use an external database for that.

If you think this partial virtual scheme would be a good fit for your
system, I'll be happy to show you how to do it.

Cheers,
Andy


More information about the ejabberd mailing list