faster image load?

Mike Driscoll wrote:

Check out Andrea's and Peter's ThumbnailCtrl: http://xoomer.alice.it/infinity77/main/ThumbnailCtrl.html

Which is nice, but I"m not sure it would solve the OP's problem.

I suspect that it takes a while to load up the entire image, then convert it to a thumbnail.

With jpeg, is is possible to load up a lower resolution image without loading the whole thing. Essentially, what you can do is load just the average value for each 8x8 pixel block -- a trick that takes advantage of how jpeg is compressed.

I doubt wx supports this, but PIL (or another image processing tool for Python) might.

Otherwise, what you would do ideally is cache the thumbnails somewhere, so you don't need to re-create them each time the app is used.

image = wx.Image(name, wx.BITMAP_TYPE_JPEG)
scaledImage = Scale(image, sizeX, sizeY)

Where does this "Scale" function from from? You might try:

scaledImage = image.Scale(sizeX, sizeY)
or
scaledImage = image.Scale(sizeX, sizeY, wx.IMAGE_QUALITY_HIGH)

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Christopher Barker wrote:

<div class="moz-text-flowed" style="font-family: -moz-fixed">Mike Driscoll wrote:

Check out Andrea's and Peter's ThumbnailCtrl: http://xoomer.alice.it/infinity77/main/ThumbnailCtrl.html

Which is nice, but I"m not sure it would solve the OP's problem.

Good point. I don't do much with images at the moment, so that's the best I could offer.

I suspect that it takes a while to load up the entire image, then convert it to a thumbnail.

With jpeg, is is possible to load up a lower resolution image without loading the whole thing. Essentially, what you can do is load just the average value for each 8x8 pixel block -- a trick that takes advantage of how jpeg is compressed.

I doubt wx supports this, but PIL (or another image processing tool for Python) might.

Otherwise, what you would do ideally is cache the thumbnails somewhere, so you don't need to re-create them each time the app is used.

image = wx.Image(name, wx.BITMAP_TYPE_JPEG)
scaledImage = Scale(image, sizeX, sizeY)

Where does this "Scale" function from from? You might try:

scaledImage = image.Scale(sizeX, sizeY)
or
scaledImage = image.Scale(sizeX, sizeY, wx.IMAGE_QUALITY_HIGH)

-Chris

I'll have to remember this...the answer is ask Chris (or Andrea or...)!

Mike

Jacek Poplawski wrote:

If anyone is interested in the solution (I found it by reading ThumbnailCtrl.py):
# from pil
import Image

...

            # fast load
            pil_image = Image.open(full)
            pil_image.thumbnail((tx,ty))

I think that PIL does not load the file with the open call, but waits until in needs to do something with it. The thumbnail() call then knows tat it doesn't need to load the full resolution of the file, and can optimize that.

PIL really is a nice lib.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

I took some advice today, and changed parts of my program.

I did alot of reading, and learned a great deal. I even took notes.

Which is better?

The log entry from the log file I'm trying to manipulate looks like this:

[Thu Jul 17 13:24:35 2008] You say, 'This is what a log entry looks like.'

And to get JUST the text and not the date stamp I was doing this:

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
linesplit = line.split('] ', 1)
lineread = linesplit[-1]
</CODE>

This would get the "You say, 'this is what a log entry looks like.' part
of the split. It was causing errors...

This is what I dug up doing some research on string manipulation,
some from Programming Python 3rd Edition by Mark Lutz, and some
from Python Phrasebook by Brad Dayley, and the rest from Python
mailing list.

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
lineread = line[27:]
</CODE>

Both accomplish the same thing. Very neat thing there the [:xx] or [xx:]

So the 2nd way takes less code, from 3 lines to 2... not as cool as the
35 lines to 6, but still less to deal with.

No lists to take parts of, and if there is an anomoly in the date/time stamp
it shouldn't throw an error unless the line is empty, but there is always a
date/time stamp on the line which is 27 chars long

So which is better to you?

The following code is part of an wx.Timer event that repeats ever second:

<CODE>
    def ProcessLog(self, event):
        global FirstLineCount
        global CurrentLineCount
        CurrentLineCount = len(open(LogDirValue, 'rb').readlines())
        if CurrentLineCount == FirstLineCount:
            CurrentTime = time.localtime(time.time())
            TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
            self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line count equal...\n')
            return
        elif CurrentLineCount > FirstLineCount:
            CurrentTime = time.localtime(time.time())
            TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
            self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line count greater...\n')
            self.FLCplus1()

        FirstLineCount = CurrentLineCount

    def FLCplus1(self):
        LR1 = linecache.getline(LogDirValue, CurrentLineCount)
        if len(LR1) == len(LR1.rstrip('\n')):
            CurrentTime = time.localtime(time.time())
            TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
            self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line count greater but not complete...\n')
            return

        LineRead1 = LR1[27:] # <-------- Here the line where the str is sliced instad of split
        for i in range(1,7):
            if LineRead1.find(globals()['C%iSearchString'%i]) >=0:
                Publisher.sendMessage(('C%i'%i, 'message'), [LineRead1])
                CurrentTime = time.localtime(time.time())
                TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
                self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Got match! Publishing...\n')

        CurrentTime = time.localtime(time.time())
        TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
        self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Done restarting...\n')
        linecache.clearcache()
</CODE>

···

----- Original Message ----- From: "Ned Batchelder" <ned@nedbatchelder.com>
To: <wxpython-users@lists.wxwidgets.org>
Sent: Thursday, 17 July, 2008 21:23
Subject: Re: [wxpython-users] split a string or [xx:] that is the question...

It's difficult to evaluate the two techniques without seeing more data. On the line you gave, both techniques should work. But you say that the first method was giving errors. Why? What was different about the lines? We'd really need to understand the full range of input strings you'll be dealing with to judge which technique is best.

--Ned.
http://nedbatchelder.com

Steve Freedenburg wrote:

I took some advice today, and changed parts of my program.

I did alot of reading, and learned a great deal. I even took notes.

Which is better?

The log entry from the log file I'm trying to manipulate looks like this:

[Thu Jul 17 13:24:35 2008] You say, 'This is what a log entry looks like.'

And to get JUST the text and not the date stamp I was doing this:

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
linesplit = line.split('] ', 1)
lineread = linesplit[-1]
</CODE>

This would get the "You say, 'this is what a log entry looks like.' part
of the split. It was causing errors...

This is what I dug up doing some research on string manipulation,
some from Programming Python 3rd Edition by Mark Lutz, and some
from Python Phrasebook by Brad Dayley, and the rest from Python
mailing list.

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
lineread = line[27:]
</CODE>

Both accomplish the same thing. Very neat thing there the [:xx] or [xx:]

So the 2nd way takes less code, from 3 lines to 2... not as cool as the
35 lines to 6, but still less to deal with.

No lists to take parts of, and if there is an anomoly in the date/time stamp
it shouldn't throw an error unless the line is empty, but there is always a
date/time stamp on the line which is 27 chars long

So which is better to you?

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Ned Batchelder, http://nedbatchelder.com

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

You thought string slicing (that's what x[27:] is doing) was nifty?
Check out string.partition.

head, sep, tail = LineRead1.partition(']')

Basically, regardless of what you .partition(), the tail is *always*
what's to the right of the first split item. Try it out.

Now, as for why your code was failing, you can always use try/except
and print out your data when it fails,

- Josiah

···

On Thu, Jul 17, 2008 at 6:55 PM, Steve Freedenburg <stevefreedenburg@charter.net> wrote:

The following code is part of an wx.Timer event that repeats ever second:

<CODE>
  def ProcessLog(self, event):
      global FirstLineCount
      global CurrentLineCount
      CurrentLineCount = len(open(LogDirValue, 'rb').readlines())
      if CurrentLineCount == FirstLineCount:
          CurrentTime = time.localtime(time.time())
          TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
          self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line
count equal...\n')
          return
      elif CurrentLineCount > FirstLineCount:
          CurrentTime = time.localtime(time.time())
          TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
          self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line
count greater...\n')
          self.FLCplus1()

      FirstLineCount = CurrentLineCount

  def FLCplus1(self):
      LR1 = linecache.getline(LogDirValue, CurrentLineCount)
      if len(LR1) == len(LR1.rstrip('\n')):
          CurrentTime = time.localtime(time.time())
          TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
          self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line
count greater but not complete...\n')
          return

      LineRead1 = LR1[27:] # <-------- Here the line where the str is
sliced instad of split
      for i in range(1,7):
          if LineRead1.find(globals()['C%iSearchString'%i]) >=0:
              Publisher.sendMessage(('C%i'%i, 'message'), [LineRead1])
              CurrentTime = time.localtime(time.time())
              TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
              self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' +
'Got match! Publishing...\n')

      CurrentTime = time.localtime(time.time())
      TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
      self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Done
restarting...\n')
      linecache.clearcache()
</CODE>

----- Original Message ----- From: "Ned Batchelder" <ned@nedbatchelder.com>
To: <wxpython-users@lists.wxwidgets.org>
Sent: Thursday, 17 July, 2008 21:23
Subject: Re: [wxpython-users] split a string or [xx:] that is the
question...

It's difficult to evaluate the two techniques without seeing more data. On
the line you gave, both techniques should work. But you say that the first
method was giving errors. Why? What was different about the lines? We'd
really need to understand the full range of input strings you'll be dealing
with to judge which technique is best.

--Ned.
http://nedbatchelder.com

Steve Freedenburg wrote:

I took some advice today, and changed parts of my program.

I did alot of reading, and learned a great deal. I even took notes.

Which is better?

The log entry from the log file I'm trying to manipulate looks like this:

[Thu Jul 17 13:24:35 2008] You say, 'This is what a log entry looks
like.'

And to get JUST the text and not the date stamp I was doing this:

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
linesplit = line.split('] ', 1)
lineread = linesplit[-1]
</CODE>

This would get the "You say, 'this is what a log entry looks like.' part
of the split. It was causing errors...

This is what I dug up doing some research on string manipulation,
some from Programming Python 3rd Edition by Mark Lutz, and some
from Python Phrasebook by Brad Dayley, and the rest from Python
mailing list.

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
lineread = line[27:]
</CODE>

Both accomplish the same thing. Very neat thing there the [:xx] or [xx:]

So the 2nd way takes less code, from 3 lines to 2... not as cool as the
35 lines to 6, but still less to deal with.

No lists to take parts of, and if there is an anomoly in the date/time
stamp
it shouldn't throw an error unless the line is empty, but there is always
a
date/time stamp on the line which is 27 chars long

So which is better to you?

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Ned Batchelder, http://nedbatchelder.com

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

Hi,

You thought string slicing (that’s what x[27:] is doing) was nifty?

Check out string.partition.

head, sep, tail = LineRead1.partition(‘]’)

Basically, regardless of what you .partition(), the tail is always

what’s to the right of the first split item. Try it out.

Most methods in this thread, including string.partition, seem to fail when there is no ‘]’ in the string. And the original method tried by Steve, splitting on ‘]’ and taking the 2nd element of the resulting list, can fail only when there’s no ‘]’ in the log-line.

Partition also doesn’t do what’s wanted when the ‘]’ is missing:

‘abcdef’.partition(‘]’)
(‘abcdef’, ‘’, ‘’)

The main part of the log-line is now in the head, not in the tail.

Splitting at position 27 doesn’t work when something about the format of the timestamp changes. (Hey, my log-analyzer broke 2 days ago because the format of the timestamp changed!! A comma was added somewhere to seperate milliseconds from the rest of the timestamp)

BTW, I don’t understand why you would want to remove a timestamp from a logfile … I often miss it in some logfiles I get!

Anyways, if you really want to take out the timestamp, I guess something like this would work:

log_line = ‘[date-time] something happened!’
ts_end_pos = log_line.find(‘]’)
if ts_end_pos >= 0:
… log_line = log_line[ts_end_pos+1:]
log_line

’ something happened!’

Myself I use regular expressions for nearly all parsing - but that’s more complexity than what’s required here.

Cheers,

–Tim

···

On Fri, Jul 18, 2008 at 7:33 AM, Josiah Carlson josiah.carlson@gmail.com wrote:

Ned,

I did :stuck_out_tongue: In the original message I put a "sample" of
what a log entry looks like.

Steve

···

----- Original Message ----- From: "Ned Batchelder" <ned@nedbatchelder.com>
To: <wxpython-users@lists.wxwidgets.org>
Sent: Friday, 18 July, 2008 06:57
Subject: Re: [wxpython-users] split a string or [xx:] that is the question...

Steve: you're asking about how to parse some text. Show us the text you are trying to parse. All of this code tells us nothing about the problem you are trying to solve. You've mentioned this challenge a few times already without actually giving us the information we need to help you.

--Ned.

Steve Freedenburg wrote:

The following code is part of an wx.Timer event that repeats ever second:

<CODE>
   def ProcessLog(self, event):
       global FirstLineCount
       global CurrentLineCount
       CurrentLineCount = len(open(LogDirValue, 'rb').readlines())
       if CurrentLineCount == FirstLineCount:
           CurrentTime = time.localtime(time.time())
           TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
           self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line count equal...\n')
           return
       elif CurrentLineCount > FirstLineCount:
           CurrentTime = time.localtime(time.time())
           TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
           self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line count greater...\n')
           self.FLCplus1()

       FirstLineCount = CurrentLineCount

   def FLCplus1(self):
       LR1 = linecache.getline(LogDirValue, CurrentLineCount)
       if len(LR1) == len(LR1.rstrip('\n')):
           CurrentTime = time.localtime(time.time())
           TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
           self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Line count greater but not complete...\n')
           return

       LineRead1 = LR1[27:] # <-------- Here the line where the str is sliced instad of split
       for i in range(1,7):
           if LineRead1.find(globals()['C%iSearchString'%i]) >=0:
               Publisher.sendMessage(('C%i'%i, 'message'), [LineRead1])
               CurrentTime = time.localtime(time.time())
               TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
               self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Got match! Publishing...\n')

       CurrentTime = time.localtime(time.time())
       TimeStamp = time.strftime("%H:%M.%S", CurrentTime)
       self.MainFrameTextCtrl.WriteText('(' + TimeStamp + ') ' + 'Done restarting...\n')
       linecache.clearcache()
</CODE>

----- Original Message ----- From: "Ned Batchelder" <ned@nedbatchelder.com>
To: <wxpython-users@lists.wxwidgets.org>
Sent: Thursday, 17 July, 2008 21:23
Subject: Re: [wxpython-users] split a string or [xx:] that is the question...

It's difficult to evaluate the two techniques without seeing more data. On the line you gave, both techniques should work. But you say that the first method was giving errors. Why? What was different about the lines? We'd really need to understand the full range of input strings you'll be dealing with to judge which technique is best.

--Ned.
http://nedbatchelder.com

Steve Freedenburg wrote:

I took some advice today, and changed parts of my program.

I did alot of reading, and learned a great deal. I even took notes.

Which is better?

The log entry from the log file I'm trying to manipulate looks like this:

[Thu Jul 17 13:24:35 2008] You say, 'This is what a log entry looks like.'

And to get JUST the text and not the date stamp I was doing this:

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
linesplit = line.split('] ', 1)
lineread = linesplit[-1]
</CODE>

This would get the "You say, 'this is what a log entry looks like.' part
of the split. It was causing errors...

This is what I dug up doing some research on string manipulation,
some from Programming Python 3rd Edition by Mark Lutz, and some
from Python Phrasebook by Brad Dayley, and the rest from Python
mailing list.

<CODE>
line = linecache.getline(logdirpath, lastlineinfile)
lineread = line[27:]
</CODE>

Both accomplish the same thing. Very neat thing there the [:xx] or [xx:]

So the 2nd way takes less code, from 3 lines to 2... not as cool as the
35 lines to 6, but still less to deal with.

No lists to take parts of, and if there is an anomoly in the date/time stamp
it shouldn't throw an error unless the line is empty, but there is always a
date/time stamp on the line which is 27 chars long

So which is better to you?

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Ned Batchelder, http://nedbatchelder.com

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Ned Batchelder, http://nedbatchelder.com

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

Steve,

Steve Freedenburg wrote:

This program at the momment will be used for a game. In the game there isn't a time stamp for the game information. And the "objective" is to look as close to the games syntex as possible. So stripping the time
stamp is desirable. So in this case, I want the information being written to the log file. I'm taking it back
out of the log file in real time. So whatever the date/time is, I know it's current, so I can take it out.
If it was some vital log that monitors server status or some other critical IT thing you're right, i'd leave it alone.
BTW I added another configuration setting to the program, where the user inputs a string for the chars to
slice off the front of the log. Maybe they want to play another game that puts a time/date stamp on the
log file, and it's only 13 chars long. They can input 13, and poof it's gone, or 0 and the time/date stamp
stays.
I thought it was a good idea anyhow. I eliminate the need for the "]" char to be there at all, and as long as
the line is 28 chars long no problems, and with a time/date stamp there, it always will be, unless, like in your
case the format was changed... Which won't be impossible to deal with.

That depends:), you might want to look into pyparsing (http://pyparsing.wikispaces.com/). Obviously one still needs to define what the log file looks like, but it provides more flexibility.

See the attached, this still works if you add e.g. milli seconds to some of the entries.

Werner

parseLogEntry.py (1.62 KB)

logfile.txt (393 Bytes)

Josiah Carlson wrote:

You thought string slicing (that's what x[27:] is doing) was nifty?
Check out string.partition.

head, sep, tail = LineRead1.partition(']')

Basically, regardless of what you .partition(), the tail is *always*
what's to the right of the first split item. Try it out.
  
Well, "try it out" if you are running at least Python 2.5. String.partition is quite a recent invention.

···

--
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

Python 2.5 came out in October 2006, and Steve started using Python
recently. And he's probably using it on Windows (which doesn't come
with Python), so it's a fair bet he's using 2.5 .

- Josiah

···

On Fri, Jul 18, 2008 at 10:54 AM, Tim Roberts <timr@probo.com> wrote:

Josiah Carlson wrote:

You thought string slicing (that's what x[27:] is doing) was nifty?
Check out string.partition.

head, sep, tail = LineRead1.partition(']')

Basically, regardless of what you .partition(), the tail is *always*
what's to the right of the first split item. Try it out.

Well, "try it out" if you are running at least Python 2.5. String.partition
is quite a recent invention.

I would make the claim that if your lines are of the following format:

[ date ] message

Then any line without a ']', which Steve wanted to get rid of along
with everything to the left, is 1) a partial line, and 2) he didn't
want that stuff on the left anyways. So it actually *does* do the
right thing.

- Josiah

···

On Fri, Jul 18, 2008 at 12:33 AM, Tim van der Leeuw <tnleeuw@gmail.com> wrote:

Hi,

On Fri, Jul 18, 2008 at 7:33 AM, Josiah Carlson <josiah.carlson@gmail.com> > wrote:

You thought string slicing (that's what x[27:] is doing) was nifty?
Check out string.partition.

head, sep, tail = LineRead1.partition(']')

Basically, regardless of what you .partition(), the tail is *always*
what's to the right of the first split item. Try it out.

Most methods in this thread, including string.partition, seem to fail when
there is no ']' in the string. And the original method tried by Steve,
splitting on ']' and taking the 2nd element of the resulting list, can fail
only when there's no ']' in the log-line.

Partition also doesn't do what's wanted when the ']' is missing:

'abcdef'.partition(']')

('abcdef', '', '')

The main part of the log-line is now in the head, not in the tail.

Splitting at position 27 doesn't work when something about the format of the
timestamp changes. (Hey, my log-analyzer broke 2 days ago because the format
of the timestamp changed!! A comma was added somewhere to seperate
milliseconds from the rest of the timestamp)

BTW, I don't understand why you would want to _remove_ a timestamp from a
_logfile_ ... I often miss it in some logfiles I get!

Anyways, if you really want to take out the timestamp, I guess something
like this would work:

log_line = '[date-time] something happened!'
ts_end_pos = log_line.find(']')
if ts_end_pos >= 0:

... log_line = log_line[ts_end_pos+1:]

log_line

' something happened!'

Myself I use regular expressions for nearly all parsing - but that's more
complexity than what's required here.