application hangs when large no of message is publish to a Topic in pubsub

Hello,

I have implemented pubsub in my app. Here is what is happening:

  1. Created a new Topic says “logs” at the start of application.

  2. From various part of my application I am sending(Publishing) message to Topic…[ almost every second]

  3. There is just one Listener which gets all the messages and dump to a GUI frame and also write those messages to a physical file (logifile)

Everything works fine when application runs for smaller time…but when application keep on running for longer time…application seems to be not responding and logfile is stopped writing …seems like pubsubs is choked when larger no of message(big message) is sent…

Can somebody help me in improving my code…I also want to know how the message is deliverer to listener when multiple publisher published messages ? what happens when they publish message at the same time…

Everything works fine when application runs for smaller time..but when application keep on running for longer time..application seems to be not responding and logfile is stopped writing ..seems like pubsubs is choked when larger no of message(big message) is sent...

Do you dump each and everything to the GUI frame which then will contain more and more text?
Or do you truncate the stored and displayed text? If you do not truncate, that might by your problem.

Can somebody help me in improving my code...I also want to know how the message is deliverer to listener when multiple publisher published messages ? what happens when they publish message at the same time..

Without code it's difficult to improve the code...

Regards,

Dietmar

···

Am 01.12.2015 um 20:29 schrieb LinuxnewBee:

Thanks…Yes I dump everything to frame and do not truncate anything…yes agreed this can add on to the performance issue…Do you have suggestion how it can be done on frame at runtime…

by “Can somebody help me in improving my code” I mean in logic, sorry for mis-stating it…

···

On Tuesday, December 1, 2015 at 3:07:03 PM UTC-5, Dietmar Schwertberger wrote:

Am 01.12.2015 um 20:29 schrieb LinuxnewBee:

Everything works fine when application runs for smaller time…but when
application keep on running for longer time…application seems to be
not responding and logfile is stopped writing …seems like pubsubs is
choked when larger no of message(big message) is sent…

Do you dump each and everything to the GUI frame which then will contain
more and more text?

Or do you truncate the stored and displayed text? If you do not
truncate, that might by your problem.

Can somebody help me in improving my code…I also want to know how
the message is deliverer to listener when multiple publisher published
messages ? what happens when they publish message at the same time…

Without code it’s difficult to improve the code…

Regards,

Dietmar

Well, that depends on your implementation.
You may just display the last few pages of logs. If the rate of
messages is high, just update every second or so.
For a start, comment out the GUI display update and make sure the
application does not keep everything in memory. Just write to the
file instead.
I have never used the wx pubsub module, but your logic seems OK to
me. If there are still problems, it’s always the best to post
example code.
Regards,
Dietmar

···

Am 01.12.2015 um 21:25 schrieb
LinuxnewBee:

    Thanks..Yes I dump everything to frame and do not

truncate anything…yes agreed this can add on to the
performance issue…Do you have suggestion how it can be done on
frame at runtime…

by " Can somebody
help me in improving my code " I mean in logic, sorry
for mis-stating it…

LinuxnewBee wrote:

1. Created a new Topic says "logs" at the start of application.
2. From various part of my application I am sending(Publishing)
message to Topic....[ almost every second]
3. There is just one Listener which gets all the messages and dump to
a GUI frame and also write those messages to a physical file (logifile)

When you say "dump to a GUI frame", what do you mean? Are you using a
listbox control, or a text control, or what? I commonly use a listbox
control for log file display, because it's so easy to add one line to
the end, but you have to set a line limit and delete the older messages
to keep the resource usage reasonable. Appending to a text box can be
an expensive operation, because you have to go read the old contents,
modify them, and send them back. If you're using a text control, you
might consider keeping the text block yourself. That way, you can add
the new message yourself without reading the old contents, and you can
trim off the old messages to keep the size reasonable.

Everything works fine when application runs for smaller time..but when
application keep on running for longer time..application seems to be
not responding and logfile is stopped writing ..seems like pubsubs is
choked when larger no of message(big message) is sent...

Well, the pubsub process gets blocked because your GUI update is taking
too long.

Can somebody help me in improving my code...I also want to know how
the message is deliverer to listener when multiple publisher published
messages ? what happens when they publish message at the same time..

Incoming messages are added to a list, and there is a "message loop"
that pulls messages out of the list and distributes them.

···

--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

LinuxnewBee wrote:
>
> 1. Created a new Topic says "logs" at the start of application.
> 2. From various part of my application I am sending(Publishing)
> message to Topic....[ almost every second]
> 3. There is just one Listener which gets all the messages and dump to
> a GUI frame and also write those messages to a physical file (logifile)

When you say "dump to a GUI frame", what do you mean? Are you using a
listbox control, or a text control, or what? I commonly use a listbox
control for log file display, because it's so easy to add one line to
the end, but you have to set a line limit and delete the older messages
to keep the resource usage reasonable. Appending to a text box can be
an expensive operation, because you have to go read the old contents,
modify them, and send them back. If you're using a text control, you
might consider keeping the text block yourself. That way, you can add
the new message yourself without reading the old contents, and you can
trim off the old messages to keep the size reasonable.

Whenever Listener gets some messages it append it to the GUI. I am
using STC.StyledTextCtrl as I need to provide a search functionality to the
users..Need to check if StyledTextCtrl supports setting a line/word limit.

> Everything works fine when application runs for smaller time..but when
> application keep on running for longer time..application seems to be
> not responding and logfile is stopped writing ..seems like pubsubs is
> choked when larger no of message(big message) is sent...

Well, the pubsub process gets blocked because your GUI update is taking
too long.

> Can somebody help me in improving my code...I also want to know how
> the message is deliverer to listener when multiple publisher published
> messages ? what happens when they publish message at the same time..

Incoming messages are added to a list, and there is a "message loop"
that pulls messages out of the list and distributes them.

--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

--
You received this message because you are subscribed to the Google Groups
"wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

···

On Wed, Dec 2, 2015 at 1:03 PM, Tim Roberts <timr@probo.com> wrote:
--
Thanks and Regards
Hemadri Saxena
9086087510

It’s your job to limit the amount of text to be displayed, not that
of a control.
Any text control is just there to display a reasonable amount of
text. A standard text control is there to display e.g. some pages of
text. A StyledTextCtrl is there to display an amount of text that is
comparable to something you would edit in a text editor.
A logfile is called a “log” because it’s just to be written, not to
be stored in memory.
The search functionality is up to you.
You need to adapt the data structures to your needs. Don’t try to
implement a database in a text control.

Regards,

Dietmar
···

Am 02.12.2015 um 19:32 schrieb Hemadri
Saxena:

          Whenever Listener gets some

messages it append it to the GUI. I am
using STC.StyledTextCtrl as I need to provide a search
functionality to the users…Need to check
if StyledTextCtrl supports setting a line/word limit.

Hello,

I have implemented pubsub in my app. Here is what is happening:

  1. Created a new Topic says “logs” at the start of application.
  2. From various part of my application I am sending(Publishing) message to Topic…[ almost every second]
  3. There is just one Listener which gets all the messages and dump to a GUI frame and also write those messages to a physical file (logifile)

Everything works fine when application runs for smaller time…but when application keep on running for longer time…application seems to be not responding and logfile is stopped writing …seems like pubsubs is choked when larger no of message(big message) is sent…

The size of a message does not affect performance of pubsub because no copying is involved, only pass-by-reference. The total number of messages either, because messages are not cached or buffered, they are immediately dispatched to listeners.

I also want to know how the message is deliverer to listener when multiple publisher published messages ? what happens when they publish message at the same time…

Pubsub itself is single threaded. The listeners are called in same thread as sender. In a single threaded app, only one message can be sent at a time, and next message can be published only when all listeners have processed the previous message. When sender and receiver “live” in different threads, you should queue sent messages via a sync queue specific to each listener, and have the receiving threads check for messages on these queues.

Oliver

···

On 2015-12-01 8:29 PM, “LinuxnewBee” hemadri.saxena@gmail.com wrote:

Whenever Listener gets some messages it append it to the GUI. I am
using STC.StyledTextCtrl as I need to provide a search functionality to the
users..Need to check if StyledTextCtrl supports setting a line/word limit.

It's your job to limit the amount of text to be displayed, not that of a
control.
Any text control is just there to display a reasonable amount of text. A
standard text control is there to display e.g. some pages of text. A
StyledTextCtrl is there to display an amount of text that is comparable to
something you would edit in a text editor.
A logfile is called a "log" because it's just to be written, not to be
stored in memory.
The search functionality is up to you.
You need to adapt the data structures to your needs. Don't try to
implement a database in a text control.

Regards,

Dietmar

I think you are not understanding what I am trying to say here, let try one
more time...
In my application I have few senders and just one listener.. Listener
listens for messages from all senders and whenever it gets a message it
does two task---one it append it to StyledTextCtrl and second write it to
physical text file says "logFile.txt".

Now on GUI side I have also provide a search functionality to user, where
user can do a local search for text within the StyledTextCtrl displayed
text only.

I was under impression that StyledTextCtrl is will do the truncation of
the text as it gets overflow.

Regarding your comment:
That's what I want to know what is "reasonable amount of text" that
StyledTextCtrl can display and what happens when that "reasonable amount
of text" exceeds . What do you mean by "Don't try to implement a database
in a text control." ?

···

On Wed, Dec 2, 2015 at 5:52 PM, Dietmar Schwertberger < maillist@schwertberger.de> wrote:

Am 02.12.2015 um 19:32 schrieb Hemadri Saxena:

--
You received this message because you are subscribed to the Google Groups
"wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Thanks and Regards
Hemadri Saxena
9086087510

>
> Hello,
>
> I have implemented pubsub in my app. Here is what is happening:
>
> 1. Created a new Topic says "logs" at the start of application.
> 2. From various part of my application I am sending(Publishing) message
to Topic....[ almost every second]
> 3. There is just one Listener which gets all the messages and dump to a
GUI frame and also write those messages to a physical file (logifile)
>
>
> Everything works fine when application runs for smaller time..but when
application keep on running for longer time..application seems to be not
responding and logfile is stopped writing ..seems like pubsubs is choked
when larger no of message(big message) is sent...

The size of a message does not affect performance of pubsub because no
copying is involved, only pass-by-reference. The total number of messages
either, because messages are not cached or buffered, they are immediately
dispatched to listeners.

Thanks that's an important piece of information

> I also want to know how the message is deliverer to listener when
multiple publisher published messages ? what happens when they publish
message at the same time..

Pubsub itself is single threaded. The listeners are called in same thread
as sender. In a single threaded app, only one message can be sent at a
time, and next message can be published only when all listeners have
processed the previous message. When sender and receiver "live" in
different threads, you should queue sent messages via a sync queue specific
to each listener, and have the receiving threads check for messages on
these queues.

ahh...for me some of my senders and receiver "live" in different threads
and few senders are in same thread as receiver. So for all those senders
which are in different thread. I will create a queue and push the messages
to queue then do I need pass whole queue itself ? For all those senders
which in same thread as that of receiver it should be as it is.

···

On Wed, Dec 2, 2015 at 9:59 PM, oliver <oliver.schoenborn@gmail.com> wrote:

On 2015-12-01 8:29 PM, "LinuxnewBee" <hemadri.saxena@gmail.com> wrote:

Oliver

--
You received this message because you are subscribed to the Google Groups
"wxPython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Thanks and Regards
Hemadri Saxena
9086087510

Do you expect to do that while new log lines are being generated?
It’s difficult to do a text search while a text box is frantically
scrolling. You would have to decide whether to freeze the scroll
while searching. If you don’t need to search until the procedure is
complete, you might consider just including a copy of a capable text
editor like Notepad++, and let your users use that to search.
No. If the underlying operating system has a limit, it will
truncate the text at the end, which is usually not what you want in
a log file reader.
Think about your users. If you have a megabyte of log file, which
might be 50,000 lines, is any human being ever going to want to
scroll through that? Absolutely not. They might be interested in
the last 100 lines, or in lines that pass through a filter, or in
saving to a file where they can search with tools like grep, but it
is expensive and pointless to put tens of thousands of log lines
into a text control.
The DebugView tool from SysInternals, which displays the Windows
kernel debug output, does what you are doing. They use a ListView
control for their log window, in part because it allows columns.
They can handle multiple megabytes of log messages, but they have an
option to freeze the scrolling while you searches. They also allow
filtering based on phrases, to reduce the volume of the spew.

···

Hemadri Saxena wrote:

          I think you are

not understanding what I am trying to say here, let try
one more time…
In my application I have few
senders and just one listener… Listener listens for
messages from all senders and whenever it gets a message
it does two task—one it append it to StyledTextCtrl
and second write it to physical text file says
“logFile.txt”.

            Now on GUI side I have also

provide a search functionality to user, where user can
do a local search for text within the StyledTextCtrl
displayed text only.

            I was under impression that

StyledTextCtrl is will do the truncation of the text as
it gets overflow.

Regarding your comment:

            That's what I want to know what

is “reasonable amount of text” that StyledTextCtrl can
display and what happens when that “reasonable amount of
text” exceeds .

-- Tim Roberts, Providenza & Boekelheide, Inc.

timr@probo.com

> I also want to know how the message is deliverer to listener when

multiple publisher published messages ? what happens when they publish
message at the same time..

Pubsub itself is single threaded. The listeners are called in same thread
as sender. In a single threaded app, only one message can be sent at a
time, and next message can be published only when all listeners have
processed the previous message. When sender and receiver "live" in
different threads, you should queue sent messages via a sync queue specific
to each listener, and have the receiving threads check for messages on
these queues.

ahh...for me some of my senders and receiver "live" in different threads
and few senders are in same thread as receiver. So for all those senders
which are in different thread. I will create a queue and push the messages
to queue then do I need pass whole queue itself ?

Well, possibly. You only need the queue on the receiver side. However, one
nice aspect of pubsub is that each listener is unaware of any other
listeners, and directly gets called when a sender emits a message. It would
be nice to keep this. But I think in this case you need one queue per
listener, in which case it is simpler to create a listener wrapper that
contains a queue. Something like this:

from threading import queue

class QueueingListener:
    def __init__(self, listener):
         self.__listener = listener
         self.__queue = queue
    def __call__(self, *args, **kwargs):
         self.__queue.push((args, kwargs))
    def proc_queued(self):
         # call the listener for each message queued
         while self.__queue:
               args, kwargs = self.__queue.pop()
               self.__listener(*args, **kwargs)

class SomeObj:
    def listener(self):
        print('hi')

class SomeThread(thread):

    ...

    # start running:
    ...
    obj = SomeObj()
    self.queueing_listener = QueueingListener(obj.listener)
    pub.subscribe(self.queueing_listener)

    # enter some long-running loop (download file, etc)
    while ...:
        ...do a bit of stuff
        self.queueing_listener.proc_queued()

This will ensure that if pubsub messages are received while the thread is
doing a chunk of work, the listener can be called safely when the thread
decides it is ok (ie between chunks of work). Note pubsub doesn't store
strong references to listeners (so that listeners are autmoatically
disposed of when there are no other references left to them), this means
you have to store a strong reference to the listener wrapper.

HTH.

For all those senders which in same thread as that of receiver it should
be as it is.

Correct.
Oliver

···

On Thu, Dec 3, 2015 at 10:29 AM, Hemadri Saxena <hemadri.saxena@gmail.com> wrote: