PIL and numpy
When I ran this program on my data files, I found the processing time is around 6 seconds, the memory consumption size is 230MB on a 1024x1024 size image. When I processed images resolution of 3840x2160, it took 263 seconds and 2.3 GB memory is consumed. The difference of these resolutions makes only eight times different number of pixels. But the processing time is increased more than 40 times. In my program I only use three buffers for processing, my first estimated minimal program sizes are 10MB for 1024x1024 resolution and 72MB for 3840x2160 resolution. However, the `top' reported 30 times more memory size.
When I profiled the program, the most of the time is consumed by the tuple construction (RBG value) and abs function. Therefore, I tried to use numpy to vectorize these code. A table below shows the result. My test environment of Intel Core i7-2720 2.20GHz Linux (Kubuntu 12.10, kernel 3.5.0-27), Python 2.7.
+-----------+--------------------------------------+ | image res | 1024x1024 | 3840x2160 | +-----------+-------+----------+--------+----------+ | | mem | time | mem | time | +-----------+-------+----------+--------+----------+ | native | 230MB | 6.0 sec | 2300MB | 263 sec | | numpy | 110MB | 0.21 sec | 320MB | 1.18 sec | +-----------+-------+----------+--------+----------+The performance was improved 30 times and up to 200 times faster. The memory consumption size reduced to 50% up to 15%. Actually, my first implementation can be improved only twice faster, so I was disappointed. After profiling, I found the sum function spend most of the time. I used the sum function to count the non-zero elements in the array. This sum function is python build-in function and can access to the numpy's array. However, I expect this sum function accesses to the each data and return to the python environment. When I replaced this sum with numpy.sum, the numpy.sum executed almost no time. I achieved 200 times better performance. This is pretty much like to matlab programming. (numpy is a matlab's Python port. I mean it is similar not only the syntax, but it is also similar to how to get the performance.)
ImgCompNumpy.py code
Comments