Evaluating HOPE

HOPE is a new Python "JIT compiler" developed at ETH Zurich that targets astrophysical computations. It is not universal, it works on a subset of Python language and it actually parses Python code to build AST representation, runs some optimizations, and then translates it to C++ code.

In short, HOPE allows you to decorate a pure Python function that is then going to be JIT-transpiled to C++, compiled and executed as native code.

There are various restrictions on what is actually supported. You need to hold-off some dynamic features. Function overloading is supported, but returns must be a single type. And you can only call pure Python functions (that would be also transpiled and inlined) plus a few functions from NumPy that are explicitly supported in HOPE.

What is great is that it allows direct indexing of NumPy arrays, so you can use them like very fast Python lists, without a function call penalty. For example, if you have code like this, where variable bodies is a two dimensional NumPy array ...

def advance_hope(
    bodies
  , size):

  for i in xrange(size):
    for j in xrange(i+1, size):
      dx = bodies[i,2] - bodies[j,2]

... HOPE will translate it to:

inline void advance_hope_Dd2J(
    PyObject * pbodies, npy_intp const * __restrict__ sbodies, npy_double * __restrict__ cbodies
  , npy_int64 csize) {

  for (npy_intp ci = 0; ci < csize; ++ci) {
    for (npy_intp cj = (ci + 1); cj < csize; ++cj) {
      npy_double cdx = npy_double();
      cdx = -cbodies[ cj*sbodies[1] + 2 ] + cbodies[ ci*sbodies[1] + 2 ];

Authors wrote a paper titled HOPE: A Python Just-In-Time compiler for astrophysical computations where there are multiple benchmarks from the Julia Language project comparing HOPE to CPython, PyPy, Numba and others.

I also wanted to see results myself, so I used a nbody sample from The Computer Language Benchmarks Game. Simply coded in Python we have nbody.py.

What does it take to adapt it for HOPE? Well, not that much. We need to pack data as NumPy array and extract a function that can be decorated with @hope.jit decorator.

def advance_hope(N, dt, bodies, size):
    ...
    bodies_ndarray = np.array([(b.x, b.y, b.z, b.vx, b.vy, b.vz, b.mass) for b in bodies])
    advance_hope(N, dt, bodies_ndarray, size)

@hope.jit
def advance_hope(N, dt, bodies, size):
  ...

Benchmark

This is the average timing of 3 runs with 500 000 and 5 000 000 iterations. HOPE was run with a hot JIT cache. I am running GNU/Linux (Fedora 20) on Intel i7-4770, with Python 2.7.5 and Hope 0.3.0 from PyPI. PyPy version is 2.2.1, and gcc is 4.8.3.

Iterations500 0005 000 000
HOPE0.15s0.67s
PyPy0.41s3.52s
Python9.01s91.84s

shedskin is another Python to C++ optimizing transpiler that is targeting more general use. It can not work just in time so the development process is not as seamless as with HOPE but it can run more general Python code. It supports more functions from common modules, but not NumPy. What is great is that it can pass lists and user defined objects as arguments so it can run nbody.py example without any modifications! Compiled with shedskin (and using -O3 -ffast-math) running times are 0.07s and 0.68s which is very comparable to HOPE.

Results are for illustration, you should run the code on your system before making conclusions.
It is as simple as pip install hope.
Heh, install hope :)


PS - There is a great blog article explaining why these compilers are so much faster than Python, Just-in-time compilers for number crunching in Python by Alex Rubinsteyn.

Alex is the author of a very similar JIT-compiler called Parakeet that translates Python to C. Generated code is more verbose but comparable to HOPE and it also works on unboxed NumPy arrays. It was first published in Rubinsteyn, Alex, et al. "Parakeet: a just-in-time parallel accelerator for python." Proceedings of the 4th USENIX conference on Hot Topics in Parallelism. USENIX Association, 2012. (LLVM code backend is now deprecated).


# english | October 19, 2014 3:03 AM | Komentari

Understanding Docker volumes

Docker data volumes are simply directories on the host, mounted read-write or read-only at specific paths inside the container. We can also say container "exposes" a path as the volume or that it binds the path with the volume. Some say container has a volume but that does sound a bit ambiguous.

Volumes allow container to persist data on the host, outside the union file system. Changes to files in the exposed volumes are therefore applied directly just as we would make them to the directory on the host.



path3857.png

Command "docker inspect <container>" shows all container paths mounted to volumes.

When creating a new container with "docker run -v ..." Docker mounts a volume at a given path. We can select host directory as the new volume or let the Docker make it automatically in /var/lib/docker/volumes. If we let Docker create new volume, it will automatically remove it when all containers using it are removed.



path3854.png

New container can also reuse all volumes previously associated with some other container. Command "docker run --volumes-from <container> ..." would mount all volumes associated with a given container inside a new container. This is useful for sharing data between containers where we let the Docker automatically manage where volumes are stored.

All containers are created from images. An image can provide a list of paths that should be mounted to volumes. We can see this list with "docker inspect <image>". The output only shows paths, and these paths are not mounted to any volumes yet. When we do "docker run ... " to create a new container from the image we must associate each path with a volume. If we do not give any options, Docker would automatically make required volumes in /var/lib/docker/volumes. When we remove the container, that volumes (if they are not shared with other containers) would be deleted. Each time we create new container Docker will make new volume(s) and mount them within the container.

What may look strange is that when automatically creating a volume for a new container, Docker would copy any existing data from the image. For example if we expose /var/log and create a new container without mount target, Docker would create a new volume in /var/lib/docker/volumes with a copy of /var/log from the image. If we mount volumes manually, using an empty host directory for example, nothing will be copied from the image and we will simply override mount target with a given (empty) directory.

This behavior makes it very easy to expose any path from the image as automatically managed volume to get data persistence. The new volume will be initialized with whatever data is present in the image. But it may be confusing if we manually mount the same path to a host directory that does not contain required data skeleton from the image and get a process crashing because of that.

When mounting volumes users (uid/gid) may not match and ownership and permissions might be messed up. This is why it is usually recommended to have a data-only container (without a running process) exposing volumes and a container using those volumes, both sharing same users. That saying you do not run everything as root :).


# docker, english | June 29, 2014 10:52 PM | Komentari

Try the new flat icon set for LibreOffice

Update: update2.png

There is an ongoing effort to develop the new flat icon set for LibreOffice. Please see this whiteboard wiki page for more info. The work is led by Issa Alkurtass. Icons are based on GNOME Symbolic theme and will probably be called Sifr in LibreOffice.

libo_flat_thumb.png

Building the icon set
The icons (stencils.svg) can be downloaded from the libodesign/flat-icons repository on GitHub.

In the SVG, each icon is the group or a clone with the ID property set to the icon name and the label is the same or set to just "icon". I used a Python script and Inkscape to export individual PNG files. To build the expected directory tree, the Python script will look for the same icon name in the Galaxy icon set and use the matching path when building the new set.

At first, we will need the Galaxy icon set. From the LibreOffice installation directory extract "share/config/images.zip" to the new "galaxy" directory. Next, download the most recent "stencils.svg" from the libodesign/flat-icons repository.

Now download cnvsvg.py script and run it as: python cnvsvg.py stencils.svg flat galaxy. As said, this will use Inkscape to extract individual icons to the "flat" directory, taking "galaxy" as the reference directory tree.

The flat icon set is not complete so to fill the missing icons we will use the Galaxy set again, but will also convert icons to grayscale for a better fit using ImageMagick:

for i in `find galaxy -name "*.png"`; do 
    convert $i -channel RGBA -matte -colorspace gray $i.png;
    mv $i.png $i;
done

Copy the flat icons over modified galaxy set (cp -r flat/* galaxy/) and create the new icon set archive (pushd galaxy; zip -r ../images_flat.zip *; popd).

Finally we will add this icon set to the installed LibreOffice. As we will not recompile the LibreOffice from the source code (see here for a required patch), the only option to add a new icon set is to replace an existing one. I do not like the Crystal set, so I will just copy "images_flat.zip" over "share/config/images_crystal.zip" in the LibreOffice instalation path. Restart the LibreOffice and select the Crystal set in the preferences.

If you want to try the new flat icon set quickly, download images_flat.zip that is built from "stencils.svg" as of 2013-06-04 (updated).


# english, fsn, libreoffice, tdf | May 27, 2013 4:35 AM | Komentari

Odletiš po investicije, potrošiš novac iz budžeta

ms-srbija.png

Neki bi rekli „redovna šoping tura“, mada se dešavalo da se ona izbegne i odloži, kao na primer 2006-2007. godine. Meni je tužno kako se vest o kupovini predstavlja. Stidljivo, kao da je sramota kupiti softver. Vidi se da nema ideje šta bi mi dobro mogli da uradimo sa tim softverom koji kupujemo. Sve izgleda iznuđeno, da se na kratko premosti zavisnost.

RTS tabloidno javlja da se Balmerovo vreme meri milionima. Ako su tačne prve informacije ono se sada meri sa 27 miliona dolara koje smo potrošili na nešto. Saznaće se već tačnije šta je kupljeno i koliki su ukupni troškovi kako domaći partneri budu radili ugradnju. Postupak nabavke je inicirala prethodna vlada, a realizacija je nastavljena u novom mandatu. Nisam siguran kako se državni sporazum sa jednom kompanijom uklapa u postupak javne nabavke[1], ali želim da se bavim suštinom. Državna uprava je mahom zavisna od Microsoft softvera i tokom ove godine neke licence bi verovatno morale da budu kupljene. Ima međutim i dobrih primera gde je ova zavisnost manja i gde se državni organi snalaze da optimizuju trošak softvera.

Voleo bih da postoji način da sada budu nagrađeni oni državni organi koji su odabrali alternativni i jeftiniji softver ili su licenciranje već realizovali iz sopstvenih budžeta umesto da čekaju objedinjenu nabavku.

Objedinjene nabavke „pokrivaju“ one koji o trošku softvera godinama nisu ni razmišljali. Teško da će biti prave analize potreba i merenja učinka softvera koji je stavljen u funkciju. Ranije su objedinjene nabavke rešavane po ključu, gde je svako tražio više, a mnogi na kraju dobijali manje nego što je realno potrebno. U komentarima na ovu vest postavlja se puno pitanja da li je baš na svim računarima u državnoj upravi ovaj skupi softver neophodan?

Prvi osnovni sud u Beogradu je odabrao slobodan i besplatan softver OpenOffice.org umesto komercijalnog paketa Microsoft Office. Neophodnu tehničku podršku realizuje njihova služba održavanja IT sistema, pa je trošak uključen u redovne i verovatno ne tako velike plate. Ovaj softver potpuno odgovara potrebama, a ušteda prema broju licenci nije mala. Drugima koji o softveru i mogućim uštedama možda nisu ni razmišljali država kupuje softverske licence kroz objedinjenu nabavku.

Ne znam ništa o predviđenoj dinamici i načinu podele ovih licenci korisnicima kada budu nabavljene prema sporazumu. Bilo bi fer pri podeli licenci iz objedinjene nabavke njihov trošak srazmerno preneti na budžete organa koji softver potražuju. Tako bi svako u svom domenu rada mogao da odluči postoje li veći prioriteti za trošenje novca umesto da se krije iza objedinjene nabavke.

Ivan Jelić je prokomentarisao:„Jedino što sada ima smisla raditi je insistiranje na svakom detalju dogovora, cenama, količinama, tajminzima.“. Potpuno se slažem. Ako se usput ostvari i korak ka većoj transparentnosti oko troškova softvera i načina kako se on koristi to bi moglo da bude značajno za domaće proizvođače. Da se i proizvođačima pruži šansa da izađu sa drugačijim rešenjima koja su prilagođena budžetu.

Diskusija: Ubuntu-RS forum, Fejsbuk (@grakic, @smarkovic), B92 blog...




Autor je učesnik projekta Libreofis, najpopularnijeg slobodnog kancelarijskog paketa. Preuzmite instalaciju sa interneta legalno i potpuno besplatno. Pridružite nam se na Fejsbuku, na stranici Libreofis u Srbiji.




[1] Moguće kroz poslovni odnos Microsofta i njihovih partnera koji se pojavljuju u nabavci, gde je uslovima nabavke propisan upravo softver koji proizvodi Microsoft. Nismo ovde posebni, slične javne nabavke u rascepu propisa i prakse prolaze, padaju pa se ponavljaju i u zemljama EU.


# fsn | May 12, 2013 12:10 AM | Komentari



GNOME lover
blog.goranrakic.com - Goran Rakić / Горан Ракић
Objavljeno pod uslovima licence Creative Commons Autorstvo 3.0 Srbija.
Sva prava i odgovornost za objavljene komentare zadržavaju njihovi autori.

Creative Commons License