#Do some stuff to manipulate the fields from read_file and th
Unfortunately, the user deleted his question, before i could post my suggestion. The only evidence of the question remaining is this tweet from PythonQuestions. I'm posting my extended answer here for future reference and to see, if someone more versed in python might correct me.
Major update
Please have a look at the comments section below and some of the corrections posted there. Some suggested the use of contextlib, which is a part of the standard library i didn't know about yet.. Also, it seems python >2.7 can handle more than one context at a time so that my "solution" isn't really needed.
My try
Starting off, i do not think that the syntax used is too unpythonic. If it is only used once or twice, i wouldn't change anything about it, but if someone likes to open more than one file within a single with-statement often, then writing a proxy class that adheres to the with-statements syntax might be useful. For opening two files it would look like this:
The class openTwoFiles implements the two basic methods __enter__ and __exit__ which the with-statement needs to execute. Using this class might look like this:
with openTwoFiles('test.txt','test2.txt')as(f1,f2):
print f1
print f2
Of course for the specific code in question the implementation of the csv-class and an iterator over the reader might also help, but i'm sticking with the general case, in which the opening of the files suffices. The class openTwoFiles might be extended to open a arbitrary number of files with custom flags:
One small note first, though. 2011 has totally been a python year for me. I actually discovered python for myself and for professional applications in October 2010 and have been using it since in professional and fun applications alike (actually, python makes professional applications fun, as well). The point is, i learned a lot in this year from python and pythonistiacs. Thank you all!
1. What’s the coolest Python application, framework or library you have discovered in 2011?
Let's start with cool libraries. I like Andreas Klöckners pyopencl. I have used it for some nice things over the course of the year, one of which is a parallelized triangulation algorithm for structured light projection. It has been a joy seeing the algorithm using a tiny fraction of the time it used to need and all with as little code change as possible.
The libraries i have used most often are of course numpy and scipy. scipys weave is my fallback, when i couldn't parallelize an algorithm but needed a performance push.
Lastly, i have been using mahotas and pymorph both by Luis Pedro.
Of course, there were many more libraries that i employed from time to time and I'm sure, I'm missing some really important and awesome ones, but these libraries have been very nice to me this year!
2. What new programming technique did you learn in 2011?
I wanted to report a bug to python once, where i discovered class attributes and was drawn back by how they can be used to communicate between class instances. I used this feature once for a communication class for a measurement automation application, where i had to keep an easy count of sent commands and their respective answers (to and from a remote machine). The class attribute was a dictionary containing all the class instances available with their respective id as the dictionary key. This way, it's easy to find the appropriate command object.
Another technique i did learn - to love actually - was event based programming. I used it in the same application as the class attributes for the evaluation and sorting of the tcp stack.
3. What’s the name of the open source project you contributed the most in 2011? What did you do?
Alas, i didn't have much time to contribute anywhere. I did however release some small code fragments on github and gitorious.
4. What was the Python blog or website you read the most in 2011?
I regularly read through the python planet, but i don't have a single python related blog that i read regularly.
5. What are the three top things you want to learn in 2012?
I took the machine learning course from Standford Engineering this year and would like to elaborate on the gained knowledge within python. There are some neat looking machine libraries out there that i would like to try one day.
Using large frameworks like django and such is also on my todo-list, but i didn't find any use for them yet.
Also, there are some emerging compilers for python that i would like to learn to use (Pypy, nuitka..).
6. What are the top software, app or lib you wish someone would write in 2012?
This would definitely be a comprehensive python wrapper Pointclouds (pcl). They did present a gsoc-idea for a wrapper, but it seems the project didn't take off.
Want to do your own list ? here’s how:
copy-paste the questions and answer to them in your blog
I've been fiddling with OpenCL computations for different projects in the past. One such project is the implementation of a specific triangulation algorithm for a structured light measurement system. The other one is a parallelized implementation of a 3d binary thinning algorithm, which a student of mine is writing his bachelor thesis about (in time, i hope he will be willing to publish some small description about it online..). We are both using python and the excellent module pyOpenCL by Andreas Klöckner.
As fast as OpenCL might be, to visualize the result we generally have to copy the gpu-buffered data back to the main memory and let frameworks such as Enthoughts mayavi or matplotlib or others draw it. For the end result of a computation this might be a viable solution but if we wanted to visualize how the computation itself progresses, the visualization will ultimately get in the way.
Since the data we need resides in the memory of the graphic card (usually i would use the gpu even though there are cpu-based opencl libraries by now), i guessed the graphics card should be able to use them for the visualization without even bothering the main memory or the cpu for that matter. This article by Ian Johnson who is working on bringing OpenCL acceleration to blenders particle system - or rather create a new particle system implementation using OpenCL - got me started on the interaction between OpenGL and OpenCL, so here it goes. The complete source code will be cloneable via git in time. I would like to describe the fundmental functions i'm using here and will omit any extra code that is not needed per se.
from OpenGL importGLasgl, GLU as glu, GLUT as glut
from OpenGL.arraysimport vbo
import numpy as np
import pyopencl as cl
import scipy
importos, sys
Starting off, we have to setup the glut-interface in which the OpenGL-buffer will be shown. This can be replaced by any GUI-framework, but i guess glut is very simple to setup.
This initializes glut, sets up the display mode to support RGBA with two buffers that can be swapped (GLUT_DOUBLE, it also works with only one buffer, but this seems more stable) and enables the depth field. After that we setup the window size, position and then create it with the title "Testing..." .
After that we need to setup some callbacks for glut. The most important ones are
The first one sets up the function which draws the content of the glut window, the second one is called every 10 msecs and will be responsible for initiating the redraw.
Now we need to setup our scene. But before that we need some data, that we want to display. I chose a voxel field, gave it some color based on its position. Also, i need a numpy array with random values. Based on the random value of a voxel, i would like to let it disappear or reappear.
After creating the arrays in python they are transfered to OpenGL-memory via "Vertex Buffer Object"s (vbo). I'm still not sure, what the subsequent bind does but it is not optional anyway. It is important to mind the size of the vectors used. Here, i've used 4 dimensions for both the position (arr) and the color (colarr). The color is thereby an RGBA-color and it seems (though i'm not sure), that the fourth dimension of the position will be used as a scaling factor, so i set it to 1.0.
# tell gl to draw the arrays (with offset and number of points)
gl.glDrawArrays(gl.GL_POINTS, 0, self.size**3)
# disable the state and unbind the buffers
gl.glDisableClientState(gl.GL_VERTEX_ARRAY)
gl.glDisableClientState(gl.GL_COLOR_ARRAY)
self.arrvbo.unbind()
self.colvbo.unbind()
# we use double buffer, so we have to swap the active buffer
# and get the new buffer to the foreground
glut.glutSwapBuffers()
To start the main loop of glut, we still have to call glut.glutMainLoop().
Now to the nice part: The interaction between OpenGL and OpenCL means, that we have to manipulate OpenGL data within OpenCL. This is a little bit of a illogic paradigm, since it should be OpenCL data that we want to visualize. Well, but once you've moved beyond this slight irregularity, it's easy-peasy:
First, we setup our OpenCL environment: the context and the queue.
# set up the list of GL objects to share with opencl
self.gl_objects = [self.arr_cl, self.col_cl]
We need the list of gl_objects because they have to be acquired each time we want to use them in OpenCL and released afterwards. I'm not sure, but i think this way we ensure that OpenCL and OpenGL don't hamper with the data at the same time. With cl.Buffer we set up the randarr array within OpenCL memory. The OpenCL calculation is done in the aforementioned self.execute():
Et voila: I've used some code snippets i found here but changed the image save procedure to use scipy.misc.imsave instead of the much slower matplotlib.
Today i found out that i may use figure.on_mouse_pick(function) to get a callback on a mouse_pick on mayavi-objects and it opened up a whole new way of dealing with mayavi-scenes. Right now, i'm trying to do a semi-automatic point cloud segmentation using different criteria such as distance, gradient and curvature. Since the data i'm using this on is unstructured, the curvature thing is proving to be quite tricky.
So here is what i do, AFTER scatterplotting my points using mayavi:
One common task in projected fringe analysis is the phase extraction of a phase shift. Four phase shifted sin-waves are projected unto a surface and recorded by the camera. I'm not going to get into the details of the algorithm, but to extract the (2pi-wrapped) phase value of each pixel we have to calculate the following equation:
image# being the image with a phase shift of #pi/2 and i,j being the coordinates of the pixel in question. That can be calculated simply enough using matlab or python. But using python holds the added advantage of being able to use lower level libraries quite easy. In this case, i wanted to test the effect of the use of OpenCL (in this case using a gpu). The module PyOpenCL does the trick and i did a quick-and-dirty performance test. Here is the sample code:
plt.title('Performance Test for PyOpenCL (red*50) vs. numpy (blue)')
plt.savefig('performance.png')
plt.show()
The program simulates the time needed to calculate an acos over an image with increasing dimensions. As can be seen in the code, the time for the gpu-calculations has been multiplied by 50 just to make it visible in the plot. And here is the plot:
While the numpy-evaluation of the term features an exponential time consumption, the calculation doesn't show any scaling troubles on the gpu! (I didn't calculate the time it takes to get the images unto the gpu and getting them back.. So that is the next step. The line starting with "gputime = ..." is changed to the following:
In the next plot the 50-times scaling of gtime has been omitted:
Now the gpu-computation time also grows exponentially but by far not as fast as the numpy-calculations. That is really interesting, but i'd have to see if it is worth binding another module. The gpu calculates a 1000x1000 image roughly 8 times quicker than numpy, but this translates to a time reduction of about 0.14 seconds! I'd have to test the measurement on real data and maybe incorporate more calculations. Still, i think it's neat that i could access the gpu without any trouble.
HT: PyOpenCL has been developed and is maintained by Andreas Klöckner.
Note to self: Find out about residuals of planefits using singular value decomposition. And while you're at it: find out, how least square fitting works with singular value decomposition anyhow. Right now - as with any transformation matrix bigger than 3x3 - i mostly think it's magic..
# where B is a column vector of the plane coefficients
# in the form b(1)*X + b(2)*Y +b(3)*Z + b(4) = 0.
t = XYZ.T
p = (np.ones((npts,1)))
A = np.hstack([t,p])
if npts == 3: # Pad A with zeros
A = [A, np.zeros(1,4)]
[u, d, v] = np.linalg.svd(A)# Singular value decomposition.
#print v[3,:]
B = v[3,:];# Solution is last column of v.
nn = np.linalg.norm(B[0:3])
B = B / nn
plane = Plane(Point(B[0],B[1],B[2]),D=B[3])
return plane
The geometry-module is the one i've posted on github. It enables me to do fast transformations and - the most important part - intersections between planes and lines. Go check that out. I would like to hear some feedback..
Back to the problem: I would like to also return a measure of error or residual. If anyone could help..
Update (25.10.2010): There was a small but still significant error in the algorithm. I assumed that B[0:2] returns the elements [B[0],B[1],B[2]], but it appears the slicing in python is in line with the syntax for a "for"-statement. So, the line with the norm had to be changed to
Learning new languages can be sometimes a little bit frustrating. Mostly, because you not only have to learn the syntax and conventions of the language but mainly because initially it's hard to know, what kinds of libraries are available - and reasonable - in that specific language.
Python's syntax seems really easy. I'd still have to look out for the indentation. But the real pain is getting to know the libraries and figuring out which one to use to achieve the specific goals. Right now, i'm trying to use opencv (which i've already used with c++) and figuring out how to calibrate a projector for a structured light sensor (fringe projection actually). There are of course many ready-to-use algorithms and even source codes, but i didn't find any yet, that cater to my specific needs.
I'll try reporting on progress concerning the projector calibration, but for now, here is a problem i encountered using the python-binding package of opencv that comes with ubuntu lucid lynx. It only supports the old bindings and there are many methods missing there! An upstream bug report explains it all. When trying to bind opencv via
Of course, i could have built the wrapper myself, but there is a repository with the newest opencv library (2.1) and the right binding library for python.
For a client of mine, i've been using the PEAR-package HTML_QuickForm which is 'superseded' by now and isn't compatible to php 5.2, whereas the followup package HTML_Quickform2 is still in alpha state. Since the client switched his server and that is running php 5.2, i had to come up with a new solution. I tried out many wordpress contact-form-plugins but they all did not satisfy me fully.
The only form-plugin that i truly would recommend is cForms II. It is flexible, can handle more than one form template. Forms can be inserted via template files or via tags inside posts or pages. I think this is the only plugin that supports individual Fieldsets (this can be very handy for designing complex layouts). The verification of the user input is done with very individual regular expressions. The admin can set Form field name, tooltip and default value and even let the default value be auto-cleared once the form field is activated. cForms II supports multi-page forms (which i didn't use, though) and it can autoconfirm emails to the user (which i didn't use either). Spam detection is done not only with the individual regular expressions, but also with a captcha-plugin or alternativey with a simple Q&A-field (like, 'what color is snow?'). The possible questions for the Q&A field can be set manually. You can even backup and restore the form settings! Furthermore the admin can chose from a dozen css-layouts and manually change them.
Here are the other plugins, that i tried out:
Contact Form 7: It didn't seem to have a template tag, that i could use in the theme files.
Form Builder: Very bad design. It features div-tags that are floated. If your design uses floating objects, it will break it or be broken itself!
Scaleable Contact Form plugin: Nearly no option to verify the user input. Supports Captcha, but it uses the plugin "simple captcha" for that, which doesn't seem to support wordpress 2.9.
Spam-Free Contact Form: simply crap. don't use that.
Visitor Contact: only interesting, if you like using external web applications on your own blog.
Yes! I've managed to scavenge an old pen-plotter that was obviously gathering dust down in the basement of our institute. Last time i've seen such a plotter was in the 1980s at the institute where my father used to work. Back then, it was a paper-feed pen plotter that was used to plot various measured data online.
The DXY-990 plotter shown below can hold eight pen at the higher left part and moves on an xy-plane using two stepper motors which control a ribbon each. Each initialization process involves finding the lower-left corner which seems to be offset from the x-position of the pen. The plotter can be loaded with a paper size slightly bigger than DIN A3. The paper is held by a controllable static charge (wow! when did this technology disappear?) so that it doesn't move when the pen is hovering or scratching on it.
The plotter has a manual dial to control it. The pen can be moved in 8 directions and two velocities. A number dial is used to change the pen. Then there are the buttons for "paper hold", "pen up/down" which simply controls a electric magnet and "pause" which can pause a loaded program. To test the plotter, one can hold the "Enter" button while turning it on (back side). It will then plot a test page.
As i expected, the control of the plotter with a pc wasn't so complicated, but finding a documentation about how to do it using standard windows/linux commands took some time. First one has to know about the HPGL-language which seems to have been a standard for plotting machines by hp which was adopted in large parts by other manufacturers. Roland DG adopted the language as RD-GL1 which is compatible to HPGL. A HPGL-command to initialize and print a line from (10,10) to (1000,1000) would look somewhat like this:
IN stands for "initialize". It sets back all the parameters (scaling, rotation, pen status etc..). SP stands for "select pen" and the first parameter "1" means, that the first pen should be selected. Interestingly, the dxy-990 remembers at every state, which pen it's holding and which actualy coordinates it should be at. So SP2 while holding the first pen means actually "bring back pen1, pick up pen2, go back to coordinates". To move the pen without drawing we can use the PU or "pen up" command. It takes two parameters as in x and y coordinate. I'm not quite sure about the delimetation codex. i think it's o.k. to write PU 10,10; or even P U 10,10;. To set the pen down, the command PD is used - equally with way points. PD can take as many waypoints as need be. Printing a polygon is much better using the compact notation PD1000,1000,50,50,10,40....
The easiest way to send the commands to the plotter is to write them down into a text file and then (using windows) type the following command into a shell
This is assuming the plotter is connected to the (still existing) parallel port. I didn't do serial port connection yet. The equivalent linux command should be
After each "return", the commands are sent to the plotter and it immediately performs that command.
By the way, HPGL-files can also be generated by Inkscape. The problem is that right now, the export-plugin doesn't consider changing the pen for differently colored figures and rectangles arent't exported at all. I hope this changes in the near future (maybe i will find time to work on it). I will end with a small video of the Roland DG DXY-990 plotter in action (plotting the test page). The ogg-video can be viewed with current browsers with the html5-standard and the ogg-vorbis codec implemented . Otherwise download the video and watch it offline (vlc).
I needed to implement gray-code sequences for a fringe projection system i'm trying to build. To avoid hard to understand and not really intuitive C-code, i used gimp to produce the projection images. In this article i'm going to document the code used.
First is the script-fu function to generate a picture with a given number of fringes or - alternatively - a given width of the period:
.
