Tag Archives: Programming

Pythonic way of opening more than one file at a time

Posted on by
17

Some hours ago, someone asked on StackExchange about a more pythonic way to represent the following code:

PLAIN TEXT
PYTHON:
  1. with open(read_csv, 'r') as read_file:
  2.     with open(write_csv, 'r') as write_file:
  3.         reader = csv.reader(read_file)
  4.         writer = csv.writer(write_file)
  5.         for row in reader:
  6.             #Do some stuff to manipulate the fields from read_file and th

Unfortunately, the user deleted his question, before i could post my suggestion. The only evidence of the question remaining is this tweet from PythonQuestions. I'm posting my extended answer here for future reference and to see, if someone more versed in python might correct me.

Major update

Please have a look at the comments section below and some of the corrections posted there. Some suggested the use of contextlib, which is a part of the standard library i didn't know about yet.. Also, it seems python >2.7 can handle more than one context at a time so that my "solution" isn't really needed.

My try

Starting off, i do not think that the syntax used is too unpythonic. If it is only used once or twice, i wouldn't change anything about it, but if someone likes to open more than one file within a single with-statement often, then writing a proxy class that adheres to the with-statements syntax might be useful. For opening two files it would look like this:

PLAIN TEXT
PYTHON:
  1. class openTwoFiles():
  2.     def __init__(self, file1, file2):
  3.         self.fn1 = file1
  4.         self.fn2 = file2
  5.    
  6.     def __enter__(self):
  7.         self.f1 = open(self.fn1,'r')
  8.         self.f2 = open(self.fn2,'r')
  9.         return (self.f1, self.f2)
  10.    
  11.     def __exit__(self, type, value, traceback):
  12.         self.f1.close()
  13.         self.f2.close()

The class openTwoFiles implements the two basic methods __enter__ and __exit__ which the with-statement needs to execute. Using this class might look like this:

PLAIN TEXT
PYTHON:
  1. with openTwoFiles('test.txt','test2.txt') as (f1,f2):
  2.     print f1
  3.     print f2

Of course for the specific code in question the implementation of the csv-class and an iterator over the reader might also help, but i'm sticking with the general case, in which the opening of the files suffices. The class openTwoFiles might be extended to open a arbitrary number of files with custom flags:

PLAIN TEXT
PYTHON:
  1. class openFiles():
  2.     def __init__(self, files, flags):
  3.         if isinstance(files,basestring):
  4.             files = [files]
  5.         if isinstance(flags,basestring):
  6.             flags = [flags]
  7.         assert len(flags)==len(files)
  8.         self.files = files
  9.         self.flags = flags
  10.    
  11.     def __enter__(self):
  12.         self.fhs = []
  13.         for f, fl in zip(self.files, self.flags):
  14.             self.fhs.append(open(f,fl))
  15.         return self.fhs
  16.    
  17.     def __exit__(self, type, value, traceback):
  18.         for f in self.fhs:
  19.             f.close()
  20.  
  21. with openFiles(['test.txt','test2.txt'], ['r','r']) as ll:
  22.     print ll

Visualizing OpenCL computations within OpenGL

Posted on by
2

I've been fiddling with OpenCL computations for different projects in the past. One such project is the implementation of a specific triangulation algorithm for a structured light measurement system. The other one is a parallelized implementation of a 3d binary thinning algorithm, which a student of mine is writing his bachelor thesis about (in time, i hope he will be willing to publish some small description about it online..). We are both using python and the excellent module pyOpenCL by Andreas Klöckner.

As fast as OpenCL might be, to visualize the result we generally have to copy the gpu-buffered data back to the main memory and let frameworks such as Enthoughts mayavi or matplotlib or others draw it. For the end result of a computation this might be a viable solution but if we wanted to visualize how the computation itself progresses, the visualization will ultimately get in the way.

Since the data we need resides in the memory of the graphic card (usually i would use the gpu even though there are cpu-based opencl libraries by now), i guessed the graphics card should be able to use them for the visualization without even bothering the main memory or the cpu for that matter. This article by Ian Johnson who is working on bringing OpenCL acceleration to blenders particle system - or rather create a new particle system implementation using OpenCL - got me started on the interaction between OpenGL and OpenCL, so here it goes. The complete source code will be cloneable via git in time. I would like to describe the fundmental functions i'm using here and will omit any extra code that is not needed per se.

This is how my module import section looks like:

PLAIN TEXT
PYTHON:
  1. from OpenGL import GL as gl, GLU as glu, GLUT as glut
  2. from OpenGL.arrays import vbo
  3. import numpy as np
  4. import pyopencl as cl
  5. import scipy
  6. import os, sys

Starting off, we have to setup the glut-interface in which the OpenGL-buffer will be shown. This can be replaced by any GUI-framework, but i guess glut is very simple to setup.

PLAIN TEXT
PYTHON:
  1. glut.glutInit()
  2. glut.glutInitDisplayMode(glut.GLUT_RGBA | glut.GLUT_DOUBLE | glut.GLUT_DEPTH)
  3. glut.glutInitWindowSize(self.width, self.height)
  4. glut.glutInitWindowPosition(0, 0)
  5. self.win = glut.glutCreateWindow("Testing...")

This initializes glut, sets up the display mode to support RGBA with two buffers that can be swapped (GLUT_DOUBLE, it also works with only one buffer, but this seems more stable) and enables the depth field. After that we setup the window size, position and then create it with the title "Testing..." ;-) .

After that we need to setup some callbacks for glut. The most important ones are

PLAIN TEXT
PYTHON:
  1. glut.glutDisplayFunc(self.draw)
  2. glut.glutTimerFunc(10, self.timer, 10)

The first one sets up the function which draws the content of the glut window, the second one is called every 10 msecs and will be responsible for initiating the redraw.

Now we need to setup our scene. But before that we need some data, that we want to display. I chose a voxel field, gave it some color based on its position. Also, i need a numpy array with random values. Based on the random value of a voxel, i would like to let it disappear or reappear.

PLAIN TEXT
PYTHON:
  1. self.randarr = np.random.rand(size, size, size)
  2. self.randarr = np.require(self.randarr, 'f')
  3. arr = [(float(x)/(size-1)-.5,float(y)/(size-1)-.5,float(z)/(size-1)-.5,1.0) for x in xrange(size) for y in xrange(size) for z in xrange(size)]
  4. self.arr = np.require(arr, 'f')
  5. colarr = [(.5-float(x)/size+.2,float(y)/size-.5+.2,float(z)/size-.5+.2,1.0) for x in xrange(size) for y in xrange(size) for z in xrange(size)]
  6. self.colarr = np.require(colarr, 'f')
  7. self.arrvbo = vbo.VBO(data=self.arr, usage=gl.GL_DYNAMIC_DRAW, target=gl.GL_ARRAY_BUFFER)
  8. self.arrvbo.bind()
  9. self.colvbo = vbo.VBO(data=self.colarr, usage=gl.GL_DYNAMIC_DRAW, target=gl.GL_ARRAY_BUFFER)
  10. self.colvbo.bind()

After creating the arrays in python they are transfered to OpenGL-memory via "Vertex Buffer Object"s (vbo). I'm still not sure, what the subsequent bind does but it is not optional anyway. It is important to mind the size of the vectors used. Here, i've used 4 dimensions for both the position (arr) and the color (colarr). The color is thereby an RGBA-color and it seems (though i'm not sure), that the fourth dimension of the position will be used as a scaling factor, so i set it to 1.0.

We can go directly to the drawing function:

PLAIN TEXT
PYTHON:
  1. def draw(self):
  2.     # execute the OpenCL computation
  3.     self.execute()
  4.     # clear the gl buffers
  5.     gl.glClear(gl.GL_COLOR_BUFFER_BIT | gl.GL_DEPTH_BUFFER_BIT)
  6.     # tell opengl to use the vertex-array and the color-array
  7.     gl.glEnableClientState(gl.GL_VERTEX_ARRAY)
  8.     gl.glEnableClientState(gl.GL_COLOR_ARRAY)
  9.     # tell opengl where to find the arrays
  10.     self.arrvbo.bind()
  11.     gl.glVertexPointer(4, gl.GL_FLOAT, 0, self.arrvbo)
  12.     self.colvbo.bind()
  13.     gl.glColorPointer(4, gl.GL_FLOAT, 0, self.colvbo)
  14.     # tell gl to draw the arrays (with offset and number of points)
  15.     gl.glDrawArrays(gl.GL_POINTS, 0, self.size**3)
  16.     # disable the state and unbind the buffers
  17.     gl.glDisableClientState(gl.GL_VERTEX_ARRAY)
  18.     gl.glDisableClientState(gl.GL_COLOR_ARRAY)
  19.     self.arrvbo.unbind()
  20.     self.colvbo.unbind()
  21.     # we use double buffer, so we have to swap the active buffer
  22.     # and get the new buffer to the foreground
  23.     glut.glutSwapBuffers()

To start the main loop of glut, we still have to call glut.glutMainLoop().

Now to the nice part: The interaction between OpenGL and OpenCL means, that we have to manipulate OpenGL data within OpenCL. This is a little bit of a illogic paradigm, since it should be OpenCL data that we want to visualize. Well, but once you've moved beyond this slight irregularity, it's easy-peasy:

First, we setup our OpenCL environment: the context and the queue.

PLAIN TEXT
PYTHON:
  1. plats = cl.get_platforms()
  2. from pyopencl.tools import get_gl_sharing_context_properties
  3. self.ctx = cl.Context(properties=[
  4.     (cl.context_properties.PLATFORM, plats[0])]
  5.     + get_gl_sharing_context_properties(), devices=None)
  6. self.queue = cl.CommandQueue(self.ctx)

We then load and build our program...

PLAIN TEXT
PYTHON:
  1. f = open(filename, 'r')
  2. fstr = f.read()
  3. self.program = cl.Program(self.ctx, fstr).build()

... and bind our OpenGL buffers to OpenCL pointers:

PLAIN TEXT
PYTHON:
  1. mf = cl.mem_flags
  2. self.arrvbo.bind()
  3. self.arr_cl = cl.GLBuffer(self.ctx, mf.READ_WRITE, int(self.arrvbo.buffers[0]))
  4. self.colvbo.bind()
  5. self.col_cl = cl.GLBuffer(self.ctx, mf.READ_WRITE, int(self.colvbo.buffers[0]))
  6. self.randarr_cl = cl.Buffer(self.ctx, mf.READ_WRITE | mf.COPY_HOST_PTR, hostbuf=self.randarr)
  7. self.arrvbo.unbind()
  8. self.colvbo.unbind()
  9. self.queue.finish()
  10. # set up the list of GL objects to share with opencl
  11. self.gl_objects = [self.arr_cl, self.col_cl]

We need the list of gl_objects because they have to be acquired each time we want to use them in OpenCL and released afterwards. I'm not sure, but i think this way we ensure that OpenCL and OpenGL don't hamper with the data at the same time. With cl.Buffer we set up the randarr array within OpenCL memory. The OpenCL calculation is done in the aforementioned self.execute():

PLAIN TEXT
PYTHON:
  1. def execute(self):
  2.     self.randarr = np.random.rand(self.size, self.size, self.size)
  3.     self.randarr = np.require(self.randarr, 'f')
  4.     cl.enqueue_copy(self.queue, self.randarr_cl, self.randarr)
  5.     cl.enqueue_acquire_gl_objects(self.queue, self.gl_objects).wait()
  6.     global_size = (self.size**3,)
  7.     local_size = None
  8.     kernelargs = (self.arr_cl,
  9.                   self.col_cl,
  10.                   self.randarr_cl,
  11.                   self.dt)
  12.     self.program.mykernel(self.queue, global_size, local_size, *(kernelargs))
  13.     cl.enqueue_release_gl_objects(self.queue, self.gl_objects).wait()
  14.     self.queue.finish()

The important bit is the function cl.enqueue_acquire_gl_objects and the corresponding cl.enqueue_release_gl_objects.

So now to the cl-code. It's actually fairly simple:

PLAIN TEXT
C:
  1. __kernel void mykernel(__global float4* pos, __global float4* color, __global float* rand, float dt)
  2. {
  3.     unsigned int i = get_global_id(0);
  4.     if (rand[i] <.05) { color[i].w = 0; }
  5.     if (rand[i]> .999) { color[i].w = 1; }
  6. }

Et voila: I've used some code snippets i found here but changed the image save procedure to use scipy.misc.imsave instead of the much slower matplotlib.

The result can be seen here:

Video not there? Download it: movie.ogv.ogv

Update: repository

The code can be found here: https://gitorious.org/openclgltest/openclgltest

Convert Gray-Code to decimal/binary and back

Posted on by
Reply

Gray-Code is a binary code developed by Frank Gray in 1947. It is used by most absolute incremental encoders and some favor it's use for fringe projection systems that have to encode the pixel data in a series of image patterns (mostly fringes with differing widths). To convert Gray-Code from and to Decimal numbers using c++ i had to look up different approaches with varying difficulty levels. I then compiled the following algorithms, that i will be using for my own fringe projection system (keep tuned).

It's rather easy if you know how to do it. I had to find out the hard way, so here's for the rest of the world:

First the easy part: transforming a integer number dec into a Gray-code _code:

PLAIN TEXT
C++:
  1. void _calculateGrayCode () {
  2.     bitset<numeric_limits<int>::digits + 1> bs(dec);
  3.     bitset<numeric_limits<int>::digits + 1> gs(dec);
  4.     bitset<numeric_limits<int>::digits + 1> _code(0);
  5.     bs>>= 1;
  6.     gs ^= bs;
  7.     for (unsigned i=0; i<_code.size(); i++){
  8.         _code[i] = gs[i];
  9.         gs[i]=0;
  10.     }
  11. }

Now to the part of converting Gray-Code to binary and then decimal number: Suppose our Gray-code is inside the bitset _code.

PLAIN TEXT
C++:
  1. void _calculateNumber () {
  2.     bitset<numeric_limits<int>::digits + 1> gs(0);
  3.     bitset<numeric_limits<int>::digits + 1> bs(0);
  4.     for (unsigned i=0; i<_code.size(); i++){
  5.         gs[i] = _code[i];
  6.     }
  7.     bs[gs.size()-1] = gs[gs.size()-1];
  8.     for (int i=gs.size()-2; i>=0;i--){
  9.         bs[i]=bs[i+1];
  10.         bs[i]=bs[i]^gs[i];
  11.     }
  12.     _number = static_cast<int>(bs.to_ulong());
  13. }

bs holds the binary representation of the Gray-Code and _number is the decimal respresentation.

References:

First version of wp-stattraq is released

Posted on by
2

Some seconds ago i've uploaded the first new version of wp-stattraq to the wordpress plugin directory. I've changed the plugin to be incooperated into the administration panel and work with version 2.6 of wordpress (which it didn't up until now).

I'm planning to put more features into the plugin and squash more bugs, but right now, i'm concentrating on releasing a stable version (as in 2.6-stable) as fast as possiblem, that can be easily installed. This first version is mainly meant for those, who were running wordpress already and need this update to be able to work with wordpress 2.6.

You can follow the updates and file tickets by visiting the trac-system i set up:

http://trac.toomuchcookies.net/trac/wiki

Also, i'm planning on releasing more plugins related to wp-stattraq that contribute to seo-work.. Look forward for more to come..