Tag Archives: programmierung

New Year’s Python Meme 2012

Posted on by
2

Following Tarek Ziade's Python Meme, here's my try:

One small note first, though. 2011 has totally been a python year for me. I actually discovered python for myself and for professional applications in October 2010 and have been using it since in professional and fun applications alike (actually, python makes professional applications fun, as well). The point is, i learned a lot in this year from python and pythonistiacs. Thank you all!

1. What’s the coolest Python application, framework or library you have discovered in 2011?

Let's start with cool libraries. I like Andreas Klöckners pyopencl. I have used it for some nice things over the course of the year, one of which is a parallelized triangulation algorithm for structured light projection. It has been a joy seeing the algorithm using a tiny fraction of the time it used to need and all with as little code change as possible.

The libraries i have used most often are of course numpy and scipy. scipys weave is my fallback, when i couldn't parallelize an algorithm but needed a performance push.

Lastly, i have been using mahotas and pymorph both by Luis Pedro.

Of course, there were many more libraries that i employed from time to time and I'm sure, I'm missing some really important and awesome ones, but these libraries have been very nice to me this year!

2. What new programming technique did you learn in 2011?

I wanted to report a bug to python once, where i discovered class attributes and was drawn back by how they can be used to communicate between class instances. I used this feature once for a communication class for a measurement automation application, where i had to keep an easy count of sent commands and their respective answers (to and from a remote machine). The class attribute was a dictionary containing all the class instances available with their respective id as the dictionary key. This way, it's easy to find the appropriate command object.

The resulting code looks a little bit like this:

PLAIN TEXT
PYTHON:
  1. class command():
  2.     ids = {}
  3.     def __init__(self):
  4.         self.id = self.findemptyid()
  5.         # handle False return!
  6.         ids[self.id] = self
  7.    
  8.     def findemptyid(self):
  9.         for i in range(10000):
  10.             if i not in ids.keys():
  11.                 return i
  12.         return False

Another technique i did learn - to love actually - was event based programming. I used it in the same application as the class attributes for the evaluation and sorting of the tcp stack.

3. What’s the name of the open source project you contributed the most in 2011? What did you do?

Alas, i didn't have much time to contribute anywhere. I did however release some small code fragments on github and gitorious.

4. What was the Python blog or website you read the most in 2011?

I regularly read through the python planet, but i don't have a single python related blog that i read regularly.

5. What are the three top things you want to learn in 2012?

I took the machine learning course from Standford Engineering this year and would like to elaborate on the gained knowledge within python. There are some neat looking machine libraries out there that i would like to try one day.

Using large frameworks like django and such is also on my todo-list, but i didn't find any use for them yet.

Also, there are some emerging compilers for python that i would like to learn to use (Pypy, nuitka..).

6. What are the top software, app or lib you wish someone would write in 2012?

This would definitely be a comprehensive python wrapper Pointclouds (pcl). They did present a gsoc-idea for a wrapper, but it seems the project didn't take off.

Want to do your own list ? here’s how:

  • copy-paste the questions and answer to them in your blog
  • tweet it with the #2012pythonmeme hashtag

Visualizing OpenCL computations within OpenGL

Posted on by
2

I've been fiddling with OpenCL computations for different projects in the past. One such project is the implementation of a specific triangulation algorithm for a structured light measurement system. The other one is a parallelized implementation of a 3d binary thinning algorithm, which a student of mine is writing his bachelor thesis about (in time, i hope he will be willing to publish some small description about it online..). We are both using python and the excellent module pyOpenCL by Andreas Klöckner.

As fast as OpenCL might be, to visualize the result we generally have to copy the gpu-buffered data back to the main memory and let frameworks such as Enthoughts mayavi or matplotlib or others draw it. For the end result of a computation this might be a viable solution but if we wanted to visualize how the computation itself progresses, the visualization will ultimately get in the way.

Since the data we need resides in the memory of the graphic card (usually i would use the gpu even though there are cpu-based opencl libraries by now), i guessed the graphics card should be able to use them for the visualization without even bothering the main memory or the cpu for that matter. This article by Ian Johnson who is working on bringing OpenCL acceleration to blenders particle system - or rather create a new particle system implementation using OpenCL - got me started on the interaction between OpenGL and OpenCL, so here it goes. The complete source code will be cloneable via git in time. I would like to describe the fundmental functions i'm using here and will omit any extra code that is not needed per se.

This is how my module import section looks like:

PLAIN TEXT
PYTHON:
  1. from OpenGL import GL as gl, GLU as glu, GLUT as glut
  2. from OpenGL.arrays import vbo
  3. import numpy as np
  4. import pyopencl as cl
  5. import scipy
  6. import os, sys

Starting off, we have to setup the glut-interface in which the OpenGL-buffer will be shown. This can be replaced by any GUI-framework, but i guess glut is very simple to setup.

PLAIN TEXT
PYTHON:
  1. glut.glutInit()
  2. glut.glutInitDisplayMode(glut.GLUT_RGBA | glut.GLUT_DOUBLE | glut.GLUT_DEPTH)
  3. glut.glutInitWindowSize(self.width, self.height)
  4. glut.glutInitWindowPosition(0, 0)
  5. self.win = glut.glutCreateWindow("Testing...")

This initializes glut, sets up the display mode to support RGBA with two buffers that can be swapped (GLUT_DOUBLE, it also works with only one buffer, but this seems more stable) and enables the depth field. After that we setup the window size, position and then create it with the title "Testing..." ;-) .

After that we need to setup some callbacks for glut. The most important ones are

PLAIN TEXT
PYTHON:
  1. glut.glutDisplayFunc(self.draw)
  2. glut.glutTimerFunc(10, self.timer, 10)

The first one sets up the function which draws the content of the glut window, the second one is called every 10 msecs and will be responsible for initiating the redraw.

Now we need to setup our scene. But before that we need some data, that we want to display. I chose a voxel field, gave it some color based on its position. Also, i need a numpy array with random values. Based on the random value of a voxel, i would like to let it disappear or reappear.

PLAIN TEXT
PYTHON:
  1. self.randarr = np.random.rand(size, size, size)
  2. self.randarr = np.require(self.randarr, 'f')
  3. arr = [(float(x)/(size-1)-.5,float(y)/(size-1)-.5,float(z)/(size-1)-.5,1.0) for x in xrange(size) for y in xrange(size) for z in xrange(size)]
  4. self.arr = np.require(arr, 'f')
  5. colarr = [(.5-float(x)/size+.2,float(y)/size-.5+.2,float(z)/size-.5+.2,1.0) for x in xrange(size) for y in xrange(size) for z in xrange(size)]
  6. self.colarr = np.require(colarr, 'f')
  7. self.arrvbo = vbo.VBO(data=self.arr, usage=gl.GL_DYNAMIC_DRAW, target=gl.GL_ARRAY_BUFFER)
  8. self.arrvbo.bind()
  9. self.colvbo = vbo.VBO(data=self.colarr, usage=gl.GL_DYNAMIC_DRAW, target=gl.GL_ARRAY_BUFFER)
  10. self.colvbo.bind()

After creating the arrays in python they are transfered to OpenGL-memory via "Vertex Buffer Object"s (vbo). I'm still not sure, what the subsequent bind does but it is not optional anyway. It is important to mind the size of the vectors used. Here, i've used 4 dimensions for both the position (arr) and the color (colarr). The color is thereby an RGBA-color and it seems (though i'm not sure), that the fourth dimension of the position will be used as a scaling factor, so i set it to 1.0.

We can go directly to the drawing function:

PLAIN TEXT
PYTHON:
  1. def draw(self):
  2.     # execute the OpenCL computation
  3.     self.execute()
  4.     # clear the gl buffers
  5.     gl.glClear(gl.GL_COLOR_BUFFER_BIT | gl.GL_DEPTH_BUFFER_BIT)
  6.     # tell opengl to use the vertex-array and the color-array
  7.     gl.glEnableClientState(gl.GL_VERTEX_ARRAY)
  8.     gl.glEnableClientState(gl.GL_COLOR_ARRAY)
  9.     # tell opengl where to find the arrays
  10.     self.arrvbo.bind()
  11.     gl.glVertexPointer(4, gl.GL_FLOAT, 0, self.arrvbo)
  12.     self.colvbo.bind()
  13.     gl.glColorPointer(4, gl.GL_FLOAT, 0, self.colvbo)
  14.     # tell gl to draw the arrays (with offset and number of points)
  15.     gl.glDrawArrays(gl.GL_POINTS, 0, self.size**3)
  16.     # disable the state and unbind the buffers
  17.     gl.glDisableClientState(gl.GL_VERTEX_ARRAY)
  18.     gl.glDisableClientState(gl.GL_COLOR_ARRAY)
  19.     self.arrvbo.unbind()
  20.     self.colvbo.unbind()
  21.     # we use double buffer, so we have to swap the active buffer
  22.     # and get the new buffer to the foreground
  23.     glut.glutSwapBuffers()

To start the main loop of glut, we still have to call glut.glutMainLoop().

Now to the nice part: The interaction between OpenGL and OpenCL means, that we have to manipulate OpenGL data within OpenCL. This is a little bit of a illogic paradigm, since it should be OpenCL data that we want to visualize. Well, but once you've moved beyond this slight irregularity, it's easy-peasy:

First, we setup our OpenCL environment: the context and the queue.

PLAIN TEXT
PYTHON:
  1. plats = cl.get_platforms()
  2. from pyopencl.tools import get_gl_sharing_context_properties
  3. self.ctx = cl.Context(properties=[
  4.     (cl.context_properties.PLATFORM, plats[0])]
  5.     + get_gl_sharing_context_properties(), devices=None)
  6. self.queue = cl.CommandQueue(self.ctx)

We then load and build our program...

PLAIN TEXT
PYTHON:
  1. f = open(filename, 'r')
  2. fstr = f.read()
  3. self.program = cl.Program(self.ctx, fstr).build()

... and bind our OpenGL buffers to OpenCL pointers:

PLAIN TEXT
PYTHON:
  1. mf = cl.mem_flags
  2. self.arrvbo.bind()
  3. self.arr_cl = cl.GLBuffer(self.ctx, mf.READ_WRITE, int(self.arrvbo.buffers[0]))
  4. self.colvbo.bind()
  5. self.col_cl = cl.GLBuffer(self.ctx, mf.READ_WRITE, int(self.colvbo.buffers[0]))
  6. self.randarr_cl = cl.Buffer(self.ctx, mf.READ_WRITE | mf.COPY_HOST_PTR, hostbuf=self.randarr)
  7. self.arrvbo.unbind()
  8. self.colvbo.unbind()
  9. self.queue.finish()
  10. # set up the list of GL objects to share with opencl
  11. self.gl_objects = [self.arr_cl, self.col_cl]

We need the list of gl_objects because they have to be acquired each time we want to use them in OpenCL and released afterwards. I'm not sure, but i think this way we ensure that OpenCL and OpenGL don't hamper with the data at the same time. With cl.Buffer we set up the randarr array within OpenCL memory. The OpenCL calculation is done in the aforementioned self.execute():

PLAIN TEXT
PYTHON:
  1. def execute(self):
  2.     self.randarr = np.random.rand(self.size, self.size, self.size)
  3.     self.randarr = np.require(self.randarr, 'f')
  4.     cl.enqueue_copy(self.queue, self.randarr_cl, self.randarr)
  5.     cl.enqueue_acquire_gl_objects(self.queue, self.gl_objects).wait()
  6.     global_size = (self.size**3,)
  7.     local_size = None
  8.     kernelargs = (self.arr_cl,
  9.                   self.col_cl,
  10.                   self.randarr_cl,
  11.                   self.dt)
  12.     self.program.mykernel(self.queue, global_size, local_size, *(kernelargs))
  13.     cl.enqueue_release_gl_objects(self.queue, self.gl_objects).wait()
  14.     self.queue.finish()

The important bit is the function cl.enqueue_acquire_gl_objects and the corresponding cl.enqueue_release_gl_objects.

So now to the cl-code. It's actually fairly simple:

PLAIN TEXT
C:
  1. __kernel void mykernel(__global float4* pos, __global float4* color, __global float* rand, float dt)
  2. {
  3.     unsigned int i = get_global_id(0);
  4.     if (rand[i] <.05) { color[i].w = 0; }
  5.     if (rand[i]> .999) { color[i].w = 1; }
  6. }

Et voila: I've used some code snippets i found here but changed the image save procedure to use scipy.misc.imsave instead of the much slower matplotlib.

The result can be seen here:

Video not there? Download it: movie.ogv.ogv

Update: repository

The code can be found here: https://gitorious.org/openclgltest/openclgltest

Convert Gray-Code to decimal/binary and back

Posted on by
Reply

Gray-Code is a binary code developed by Frank Gray in 1947. It is used by most absolute incremental encoders and some favor it's use for fringe projection systems that have to encode the pixel data in a series of image patterns (mostly fringes with differing widths). To convert Gray-Code from and to Decimal numbers using c++ i had to look up different approaches with varying difficulty levels. I then compiled the following algorithms, that i will be using for my own fringe projection system (keep tuned).

It's rather easy if you know how to do it. I had to find out the hard way, so here's for the rest of the world:

First the easy part: transforming a integer number dec into a Gray-code _code:

PLAIN TEXT
C++:
  1. void _calculateGrayCode () {
  2.     bitset<numeric_limits<int>::digits + 1> bs(dec);
  3.     bitset<numeric_limits<int>::digits + 1> gs(dec);
  4.     bitset<numeric_limits<int>::digits + 1> _code(0);
  5.     bs>>= 1;
  6.     gs ^= bs;
  7.     for (unsigned i=0; i<_code.size(); i++){
  8.         _code[i] = gs[i];
  9.         gs[i]=0;
  10.     }
  11. }

Now to the part of converting Gray-Code to binary and then decimal number: Suppose our Gray-code is inside the bitset _code.

PLAIN TEXT
C++:
  1. void _calculateNumber () {
  2.     bitset<numeric_limits<int>::digits + 1> gs(0);
  3.     bitset<numeric_limits<int>::digits + 1> bs(0);
  4.     for (unsigned i=0; i<_code.size(); i++){
  5.         gs[i] = _code[i];
  6.     }
  7.     bs[gs.size()-1] = gs[gs.size()-1];
  8.     for (int i=gs.size()-2; i>=0;i--){
  9.         bs[i]=bs[i+1];
  10.         bs[i]=bs[i]^gs[i];
  11.     }
  12.     _number = static_cast<int>(bs.to_ulong());
  13. }

bs holds the binary representation of the Gray-Code and _number is the decimal respresentation.

References:

Erste Version von wp-stattraq veröffentlicht

Posted on by
1

Eben gerade habe ich die erste neue Version von wp-stattraq auf wordpress.org freigegeben. Ich habe das Plugin nur insoweit verändert, als ich es für wordpress 2.6 lauffähig gemacht habe und dabei die Ausgabe des Plugins in die normale Administrationsobefläche integriert habe.

Später möchte ich noch einige Mankos des Plugins (die ich als langzeitiger Nutzer gut kenne) beseitigen. Aber vorerst müssen einige Integrationsprobleme (wie die automatische Installation bereinigt werden).

Wer Änderungswünsche und -vorschläge hat, sollte diese bitte direkt im trac-system als ticket eintragen:

http://trac.toomuchcookies.net/trac/wiki

Auch in Planung (code ist eigentlich schon fertig) ist ein plugin, das bei der seo-optimierung helfen soll und das soll die Daten von wp-stattraq nutzen. Dazu später mehr..