PyPy related internship at NCAR
Hello everyone
I would like to advertise a PyPy-related summer internship at the National Center for Atmospheric Research, which is located in lovely Boulder, Colorado. As for the last year, the mentor will be Davide del Vento, with my possible support on the PyPy side.
The full details of the application are to be found on the internship description and make sure you read the requirements first. Important requirements:
- Must currently be enrolled in a United States university.
- Only students authorized to work for any employer in the United States will be considered for the SIParCS program.
- Must be a graduate or under graduate who has completed their sophomore year.
If you happen to fulfill the requirements, to me this sounds like a great opportunity to spend a summer at NCAR in Boulder hacking on atmospheric models using PyPy.
Cheers, fijal
Py3k status update #8
This is the eight status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.
Just a short update on November's work: we're now passing about 194 of
approximately 355 modules of CPython's regression test suite, up from passing
160 last month. Many test modules only fail a small number of individual tests
now.
We'd like to thank Amaury Forgeot d'Arc for his contributions, in particular he
has made significant progress on updating CPyExt for Python 3 this month.
Some other highlights:
-
test_marshal now passes, and there's been significant progress on
pickling (thanks Kenny Levinsen and Amaury for implementing
int.{to,from}_bytes) - We now have a _posixsubprocess module
- More encoding related fixes, which affects many failing tests
- _sre was updated and now test_re almost passes
- Exception behavior is almost complete per the Python 3 specs, what's mostly
missing now are the new __context__ and __traceback__ attributes (PEP
3134) - Fixed some crashes and deadlocks occurring during the regression tests
- We merged the unicode-strategies branch both to default and to py3k: now we
have versions of lists, dictionaries and sets specialized for unicode
elements, as we already had for strings. - However, for string-specialized containers are still faster in some cases
because there are shortcuts which have not been implemented for unicode yet
(e.g., constructing a set of strings from a list of strings). The plan is to
completely kill the shortcuts and improve the JIT to produce the fast
version automatically for both the string and unicode versions, to have a
more maintainable codebase without sacrificing the speed. The autoreds
branch (already merged) was a first step in this direction.
cheers,
Philip&Antonio
Well done. PyPy is one the most interesting projects out there today.
Keep up the amazing work guys!
J.
PyPy San Francisco Sprint Dec 1st - Dec 2nd 2012
The next PyPy sprint will be in San Francisco, California. It is a
public sprint, suitable for newcomers. It will run on Saturday December 1st and
Sunday December 2nd. The goals for the sprint are continued work towards the
2.0 release as well as code cleanup, we of course welcome any topic which
contributors are interested in working on.
Some other possible topics are:
- running your software on PyPy
- work on PyPy's numpy (status)
- work on STM (status)
- JIT improvements
- any exciting stuff you can think of
If there are newcomers, we'll run the usual introduction to hacking on
PyPy.
Location
The sprint will be held at the Rackspace Office:
620 Folsom St, Ste 100
San Francisco
The doors will open at 10AM both days, and run until 6PM both days.
Thanks to David Reid for helping get everything set up!
Hi, I'm interested in getting involved with PyPy development and would love to attend the sprint to get started, but I'm not sure I can make it both days. Would it be okay to just participate Sunday or would that not make sense?
Awww jeez, you guys couldn't wait a couple more weeks? Have fun. If anyone's still in the bay area after Dec 10th give a holler.
It would have helped a lot if this sprint was announced more in advance. I just missed it because I didn't bother to check the PyPy blog last week.
PyPy 2.0 beta 1
We're pleased to announce the 2.0 beta 1 release of PyPy. This release is not a typical beta, in a sense the stability is the same or better than 1.9 and can be used in production. It does however include a few performance regressions documented below that don't allow us to label is as 2.0 final. (It also contains many performance improvements.)
The main features of this release are support for ARM processor and compatibility with CFFI. It also includes numerous improvements to the numpy in pypy effort, cpyext and performance.
You can download the PyPy 2.0 beta 1 release here:
https://pypy.org/download.html
What is PyPy?
PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.3. It's fast (pypy 2.0 beta 1 and cpython 2.7.3 performance comparison) due to its integrated tracing JIT compiler.
This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. It also supports ARM machines running Linux. Windows 64 work is still stalling, we would welcome a volunteer to handle that.
How to use PyPy?
We suggest using PyPy from a virtualenv. Once you have a virtualenv installed, you can follow instructions from pypy documentation on how to proceed. This document also covers other installation schemes.
Regressions
Reasons why this is not PyPy 2.0:
- the ctypes fast path is now slower than it used to be. In PyPy 1.9 ctypes was either incredibly faster or slower than CPython depending whether you hit the fast path or not. Right now it's usually simply slower. We're probably going to rewrite ctypes using cffi, which will make it universally faster.
- cffi (an alternative to interfacing with C code) is very fast, but it is missing one optimization that will make it as fast as a native call from C.
- numpypy lazy computation was disabled for the sake of simplicity. We should reenable this for the final 2.0 release.
Highlights
- cffi is officially supported by PyPy. You can install it normally by using pip install cffi once you have installed PyPy and pip. The corresponding 0.4 version of cffi has been released.
- ARM is now an officially supported processor architecture. PyPy now work on soft-float ARM/Linux builds. Currently ARM processors supporting the ARMv7 and later ISA that include a floating-point unit are supported.
- This release contains the latest Python standard library 2.7.3 and is fully compatible with Python 2.7.3.
- It does not however contain hash randomization, since the solution present in CPython is not solving the problem anyway. The reason can be found on the CPython issue tracker.
- gc.get_referrers() is now faster.
- Various numpy improvements. The list includes:
- axis argument support in many places
- full support for fancy indexing
- complex128 and complex64 dtypes
- JIT hooks are now a powerful tool to introspect the JITting process that PyPy performs.
- **kwds usage is much faster in the typical scenario
- operations on long objects are now as fast as in CPython (from roughly 2x slower)
- We now have special strategies for dict/set/list which contain unicode strings, which means that now such collections will be both faster and more compact.
Things we're working on
There are a few things that did not make it to the 2.0 beta 1, which are being actively worked on. Greenlets support in the JIT is one that we would like to have before 2.0 final. Two important items that will not make it to 2.0, but are being actively worked on, are:
- Faster JIT warmup time.
- Software Transactional Memory.
Cheers,
Maciej Fijalkowski, Armin Rigo and the PyPy team
Good job! 2 things:
1) the link to the .tar.bz for Linux 64 (libc 2.13) links to a corrupted file (bz2 claims it is corrupted, and its MD5 hash doesn't match the one on the page)
2) the link to the benchmark on this page: https://speed.pypy.org/comparison/?exe=1%2B785,2%2B472&ben=1,34,27,2,25,3,46,4,5,41,42,22,44,6,39,7,8,45,23,24,9,10,11,12,13,40,14,15,35,36,37,38,16,28,30,32,29,33,17,18,19,20,43&env=1,2&hor=true&bas=2%2B472&chart=normal+bars
is empty -- no charts were plotted. (I've turned off all my adblocking).
The OSX binary segfaults on a Lion 64bit. I tried both 2.0-beta1 and a nightly build. Notice, 1.9 works perfectly.
I would be more than happy to give it a shot if there was solid PostgreSQL support - otherwise it is a no-go for me.
Just announced on the IRC channel: psycopg2cffi. They ported it for speed, but from my CFFI experience, I think the biggest advantage is maintainability.
I think I should give a try to this.
Goona give a shot
---------------
www.insecuregeek.blogspot.com
If we can get greenlet support in the JIT that'd be fantastic - my non-blocking driver for MongoDB, Motor, will need it before it's usable with PyPy. Thanks for the amazing work!
Py3k status update #7
This is the seventh status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.
The biggest news is that this month Philip started to work on py3k in parallel
to Antonio. As such, there was an increased amount of activity.
The py3k buildbots now fully translate the branch every night and run the
Python standard library tests.
We currently pass 160 out of approximately 355 modules of CPython's standard
test suite, fail 144 and skip approximately 51.
Some highlights:
- dictviews (the objects returned by dict.keys/values/items) has been greatly
improved, and now they full support set operators - a lot of tests has been fixed wrt complex numbers (and in particular the
__complex__ method) - _csv has been fixed and now it correctly handles unicode instead of bytes
- more parser fixes, py3k list comprehension semantics; now you can no longer
access the list comprehension variable after it finishes - 2to3'd most of the lib_pypy modules (pypy's custom standard lib
replacements/additions) - py3-enabled pyrepl: this means that finally readline works at the command
prompt, as well as builtins.input(). pdb seems to work, as well as
fancycompleter to get colorful TAB completions :-) - py3 round
- further tightening/cleanup of the unicode handling (more usage of
surrogateescape, surrogatepass among other things) - as well as keeping up with some big changes happening on the default branch
and of course various other fixes.
Finally, we would like to thank Amaury Forgeot d'Arc for his significant
contributions.
cheers,
Philip&Antonio
thanks for sharing.
How do I compile/translate it for testing the py3k branch?
How much optimization is already possible?
@arne: you can just use the usual translate.py command inside the py3k branch.
Or download one of the nightly builds:
https://buildbot.pypy.org/nightly/py3k/
however, note that:
- JIT is not enabled (yet)
- no focus has been put on performances (yet :)) so it is probably slower than even the non-jitted python2
@antonio: for me the translate with goal pypy (interpreter) did not work, so I asked.
I’ll try again. Thanks!
NumPy status update #5
Hello.
I'm quite excited to inform that work on NumPy in PyPy has been restarted and there has been quite a bit of progress on the NumPy front in PyPy in the past two months. Things that happened:
- complex dtype support - thanks to matti picus, NumPy on PyPy now supports complex dtype (only complex128 so far, there is work on the other part)
- big refactoring - probably the biggest issue we did was finishing a big refactoring that disabled some speedups (notably lazy computation of arrays), but lowered the barrier of implementing cool new features.
- fancy indexing support - all fancy indexing tricks should now work, including a[b] where b is an array of integers.
- newaxis support - now you can use newaxis features
- improvements to ``intp``, ``uintp``, ``void``, ``string`` and record dtypes
Features that have active branches, but hasn't been merged:
- float16 dtype support
- missing ndarray attributes - this is a branch to finish all attributes on ndarray, hence ending one chapter.
- pickling support for numarray - hasn't started yet, but next on the list
More importantly, we're getting very close to able to import the python part of the original numpy with only import modifications and running it's tests. Most tests will fail at this point, however it'll be a good start for another chapter :-)
Cheers,
fijal
Hi,
are sort() and argsort(), preferably with axis parameter, in nearest future plans?
Regards, Dmitrey.
Hi Dmitrey.
argsort (with axis) is already implemented on a branch, sort coming later (it's further in the alphabet, I'm at g now ;-)
Cape Town 2012 sprint report
Hello.
We're about to finish a PyPy sprint in Cape Town, South Africa that was one of the smallest done so far, only having Armin Rigo and Maciej Fijalkowski with Alex Gaynor joining briefly at the beginning, however also one of the longest, lasting almost 3 weeks. The sprint theme seems to be predominantly "no new features" and "spring cleaning". We overall removed about 20k lines of code in the PyPy source tree. The breakdown of things done and worked on:
-
We killed SomeObject support in annotation and rtyper. This is a modest code saving, however, it reduces the complexity of RPython and also, hopefully, improves compile errors from RPython. We're far from done on the path to have comprehensible compile-time errors, but the first step is always the hardest :)
-
We killed some magic in specifying the interface between builtin functions and Python code. It used to be possible to write builtin functions like this:
def f(space, w_x='xyz'):
which will magically wrap 'xyz' into a W_StringObject. Right now, instead, you have to write:
@unwrap_spec(w_x=WrappedDefault('xyz')) def f(space, w_x):which is more verbose, but less magical.
-
We killed the CExtModuleBuilder which is the last remaining part of infamous extension compiler that could in theory build C extensions for CPython in RPython. This was never working very well and the main part was killed long ago.
-
We killed various code duplications in the C backend.
-
We killed microbench and a bunch of other small-to-medium unused directories.
-
We killed llgraph JIT backend and rewrote it from scratch. Now the llgraph backend is not translatable, but this feature was rarely used and caused a great deal of complexity.
-
We progressed on continulet-jit-3 branch, up to the point of merging it into result-in-resops branch, which also has seen a bit of progress.
Purpose of those two branches:
- continulet-jit-3: enable stackless to interact with the JIT by killing global state while resuming from the JIT into the interpreter. This has multiple benefits. For example it's one of the stones on the path to enable STM for PyPy. It also opens new possibilities for other optimizations including Python-Python calls and generators.
- result-in-resops: the main goal is to speed up the tracing time of PyPy. We found out the majority of time is spent in the optimizer chain, which faces an almost complete rewrite. It also simplifies the storage of the operations as well as the number of implicit invariants that have to be kept in mind while developing.
-
We finished and merged the excellent work by Ronan Lamy which makes the flow object space (used for abstract interpretation during RPython compilation) independent from the Python interpreter. This means we've achieved an important milestone on the path of separating the RPython translation toolchain from the PyPy Python interpreter.
Cheers,
fijal & armin
Py3k status update #6
This is the sixth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.
The coolest news is not about what we did in the past weeks, but what we will
do in the next: I am pleased to announce that Philip Jenvey has been
selected by the PyPy communitiy to be funded for his upcoming work on py3k,
thanks to your generous donations. He will start to work on it shortly, and he
will surely help the branch to make faster progress. I am also particularly
happy of this because Philip is the first non-core developer who is getting
paid with donations: he demonstrated over the past months to be able to work
effectively on PyPy, and so we were happy to approve his application for the
job. This means that anyone can potentially be selected in the future, the
only strict requirement is to have a deep interest in working on PyPy and to
prove to be able to do so by contributing to the project.
Back to the status of the branch. Most of the work since the last status
update has been done in the area of, guess what? Unicode strings. As usual,
this is one of the most important changes between Python 2 and Python 3, so
it's not surprising. The biggest news is that now PyPy internally supports
unicode identifiers (such as names of variables, functions, attributes, etc.),
whereas earlier it supported only ASCII bytes strings. The changes is still
barely visible from the outside, because the parser still rejects non-ASCII
identifiers, however you can see it with a bit of creativity:
>>>> def foo(x): pass
>>>> foo(**{'àèìòù': 42})
Traceback (most recent call last):
File "<console>", line 1, in <module>
TypeError: foo() got an unexpected keyword argument 'àèìòù'
Before the latest changes, you used to get question marks instead of the
proper name for the keyword argument. Although this might seem like a small
detail, it is a big step towards a proper working Python 3 interpreter and it
required a couple of days of headaches. A spin-off of this work is that now
RPython has better built-in support for unicode (also in the default branch):
for example, it now supports unicode string formatting (using the percent
operator) and the methods .encode/.decode('utf-8').
Other than that there is the usual list of smaller issues and bugs that got
fixed, including (but not limited to):
- teach the compiler when to emit the new opcode DELETE_DEREF (and
implement it!)- detect when we use spaces and TABs inconsistently in the source code, as
CPython does- fix yet another bug related to the new lexically scoped exceptions (this
is the last one, hopefully)- port some of the changes that we did to the standard CPython 2.7 tests to
3.2, to mark those which are implementation details and should not be run on
PyPy
Finally, I would like to thank Amaury Forgeot d'Arc and Ariel Ben-Yehuda for
their work on the branch; among other things, Amaury recently worked on
cpyext and on the PyPy _cffi_backend, while Ariel submitted a patch to
implement PEP 3138.
Python 3.3 has some absolutely crucial fixes (finally! to Unicode). I'd go as far as to say that Python 3.3 is the first Pyton version of all that is truly suitable for the full range of internationalized apps. So I wonder a bit about the set target for the PyPy3 work being 3.2. Any chance it can be 3.2 with the 3.3 Unicode implementation?
we chose to target 3.2 because at the time 3.3 was a moving target. Now we could indeed decide to retarget 3.3, but I'm not sure it's a good idea. There is still a lot of work to be done for 3.2, and adding more features would only shift the end to later.
I know this might be an odd request but... Has the pypy team ever considered inquiring Mozilla about embedding pypy into their browser? I say this because Google is embedding a vm from chromium to support dart. I do not think this is ideal for an open web. Pypy, on the other hand, would be ideal as an open web vm! Think about it!
That sounds great!
Thank you for your work - and for keeping us up to date!
PyPy Cape Town Sprint Oct 7th - Oct 21st 2012
Hello everyone!
The next PyPy sprint will be in Cape Town, South Africa. It is a public sprint, suitable for newcomers. It starts a couple of days after PyCon South Africa, which is on the 4th and 5th of October. This is a relatively unusual sprint in that it is hosted halfway across the world from where most contributors live, so we plan to spend some time during those two weeks doing sprinting and some time doing touristy stuff. The goals for the sprint are general progress and whatever people are interested in.
Possible topics:
- PyPy release 2.0
- running your software on PyPy
- work on PyPy's numpy (status)
- work on STM (status)
- JIT improvements
- any exciting stuff you can think of
If there are newcomers, we'll run the usual introduction to hacking on PyPy.
Location
The sprint will be held either in the apartment of fijal, which is in Tamboerskloof, Cape Town, or in the offices of the Praekelt Foundation, located in Woodstock, Cape Town. The Praekelt Foundation has offered to host us, if needed.
Cape Town, as a very touristy place, has tons of accomodation ranging in quality from good to amazing. Depending on the sprint location you might need a car.
Good to Know
You probably don't need visa for South Africa -- consult Wikipedia. South Africa is a lovely place with lots of stuff to do. You can see penguins, elephants, lions and sharks all on one day (or better yet, on multiple days).
There is a wide selection of good restaurants within a reasonable distance of the sprint venue (depending on the venue, either walking or driving).
The power plug is some weird derivative of an old-english standard, but adapters are easily acquired.
Who's Coming?
If you'd like to come, please let us know when you will be arriving and leaving, as well as what your interests are. We'll keep a list of people which we'll update (or you can do so yourself if you have bitbucket pypy commit rights).
Cheers,
fijal
Why pypy is three times slower than python2.6 + psyco2 ??
# text parser:
# python2.7 - 0.94s
# python2.7 + cython - 0.73s
# pypy1.9 - 0.68s
# python2.5 + psyco1.6 - 0.31s
# python2.6 + psyco2 - 0.23s
"python2.6 + psyco2" is 3.3 times faster than pypy1.9, why ??
I found that "cStringIO" is extremely slow in pypy1.9 (almost three times slower than python2.7), I'm using a lot of cStringIO in my text parser. here is my benchmark:
import time, cStringIO
def test1():
text = '1234567890' * 1024 * 256
sio = cStringIO.StringIO()
ts = time.time()
for ch in text: sio.write(ch)
print 'ts', time.time() - ts
try:
import psyco
psyco.full()
except:
pass
test1()
test1()
test1()
# python2.7 0.45s
# psyco2 0.26s
# pypy-1.9 1.30s
You could try using StringIO instead of cStringIO. pypy can optimize that much better.
Here’s an adapted example:
------ ------ ------
import time, StringIO, cStringIO
def csio():
text = '1234567890' * 1024 * 256
sio = cStringIO.StringIO()
ts = time.time()
for ch in text: sio.write(ch)
print 'ts', time.time() - ts
def nsio():
text = '1234567890' * 1024 * 256
sio = StringIO.StringIO()
ts = time.time()
for ch in text: sio.write(ch)
print 'ts', time.time() - ts
print "cStringIO"
csio()
csio()
csio()
print "StringIO"
nsio()
nsio()
nsio()
------ ------ ------
Results for me with pypy 1.9:
$ python stringiotest.py
cStringIO
ts 0.636300086975
ts 0.63633108139
ts 0.636710882187
StringIO
ts 3.35502791405
ts 3.34557986259
ts 3.33949017525
$ bin/pypy stringiotest.py
cStringIO
ts 1.05391597748
ts 0.528824090958
ts 0.530929803848
StringIO
ts 0.359623908997
ts 0.277186870575
ts 0.273662090302
Increase the amount of iterations for even higher speedups:
text = '1234567890' * 1024 * 256 * 16
⇒
$ bin/pypy stringiotest.py
cStringIO
ts 224.367353201
ts 140.621050835
ts 140.672322035
StringIO
ts 5.80670285225
ts 4.95937395096
ts 4.82084798813
$ python stringiotest.py
cStringIO
ts 9.54650998116
ts 9.60773801804
ts 9.56916093826
StringIO
ts 47.1465728283
ts 47.145359993
ts 47.1618230343
Interestingly pypy with StringIO is twice as fast as python with cStringIO. But pypy with cStringIO is slow.
So pypy with StringIO might still require 2x as much time as python2.6+psyco2.
But remember that this compares pure python code on pypy with hand-optimized C-code+psyco.
Glad to help :)
The cool part here is that pypy allows us to replace many C-modules with nicely readable python-code and still get a fast program.
And that your custom code gets the same speedups.
in order to import StringIO as cStringIO. how to confirm my script is running pypy? not python ?
NumPy on PyPy status update
Hello everyone.
It's been a while since we posted a numpy work update, but I'm pleased to inform you that work on it has been restarted. A lot of the work has been done by Matti Picus, who is one of the newest contributors to the PyPy project. None of the work below has been merged so far, it's work in progress:
- Complex dtype support.
- Fixing incompatibilities between numpy and pypy's version.
- Refactoring numpypy to simplify the code and make it easier for new contributors.
- Reuse most of the numpy's pure python code without modifications.
Finishing this is also the plan for the next month.
Cheers,
fijal
Exciting stuff!
It would be great to see a write-up of what, if anything, still remains to be done after this merge to have full compatibility with numpy.
Once we have a better idea about the numpy's test status we'll post it. That would be probably on the next month's update report.
Great to hear that!
I’m anxious to see numpy on pypy bear fruit, so I can test it with some model of a model I experiment with.
Pypy and numpypy are just great!
I will be able to move some projects completely to pypy after Linalg implementation.
In the meanwhile I just noticed that vectorized operations as dot product in numpypy are not yet as fast as in numpy.
You can post it on https://jobs.pythonweekly.com/