Python generators with multiple threads

Sometimes when writing Python there is a CPU expensive task that needs to be done for every item in an iterator. You might go to Google and search for python multithreading, which might take you to the threading module. But that isn't very useful, so next you look to the multiprocessing module. That looks more promising, but all the examples have managers or queues or pipes or some other complicated stuff.

I want simple. All I want to do is write a generator that yields something, have that something passed to a thread or process to compute the result, then have a generator yield the result when it is ready. Sounds simple. Unfortunately is is hard to find examples of how to do this in Python.

So here are some examples!

In the example below increasing_sleeps() is the function that yields something and rand_sleep() is the function that takes a long time to process each thing yielded by increasing_sleeps(). It uses multiprocessing.dummy to use threads instead of full processes.


The results will look something like this:

Slept 0 seconds
Slept 0 seconds
Slept 1 seconds
Slept 1 seconds
Slept 4 seconds
Slept 4 seconds
Slept 3 seconds
Slept 5 seconds
Slept 1 seconds
Slept 4 seconds

Cool! But what if I need to know what input was sent to rand_sleep() so I can correlate the results with the input? Simple, just yield the input from rand_sleep(). The example below does that correlation, and also uses full Python processes instead of threads. Threads are cheaper, but there are cases where you will need to use processes to get around the Global Interpreter Lock (GIL). One big gotcha with using full processes is that Python needs to pickle the function being sent to the new process (it doesn't share memory like with threads), and it can't pickle functions or instance methods. Oddly enough it can pickle instances of custom classes, so you can make a callable class and pass that instead of a function.


The results will look something like this:

Slept 0 seconds of max 0
Slept 1 seconds of max 1
Slept 1 seconds of max 2
Slept 0 seconds of max 3
Slept 3 seconds of max 5
Slept 0 seconds of max 6
Slept 4 seconds of max 4
Slept 2 seconds of max 8
Slept 5 seconds of max 7
Slept 8 seconds of max 9

Nice! That makes it clear that the results are coming back out of order. I you want the results to comeback in order, use Pool.imap() instead of Pool.imap_unordered().

And finally, here is an example of a simple yet useful use of these methods:

Role-based Puppet with Hiera

This is a method of configuring machines with Puppet based on their roles. It's similar to Jordan Sissel's nodeless Puppet in that there are no individual node declarations, just a default node that assigns one class. That one class is responsible for including all the other classes based on what 'roles' are assigned to the node. One difference between this role-based puppet and Jordan's nodeless puppet is that I'm not using an ENC. Instead of an ENC I'm using Hiera to assign to roles and specify custom parameters for the nodes. I put together a demo of role-based puppet as a Vagrant configuration, which you can get on GitHub.

Here is what the default node looks like:


The role class searches Hiera for 'roles' and includes them:


In Hiera multiple roles can be assigned, and data can be added to customize the roles. In this case we are adding the 'role::lb' load balancer role along with 'role::base', and specifying how much memory memcached should use and how many nginx processes should run.


The roles then include the actual classes that do the work, in this case nginx and memcache. In the real world the role would probably do a lot more, like configure sysctl. The role can pass site specific parameters to the classes, but it can also rely on Puppet to lookup parameters in Hiera.

Site-specific roles and generic modules

Every site is different, but there is no reason that every company should have its own nginx module. Modules should be generic and customizable, they shouldn't make assumptions about how you want to configure something. Roles are site-specific, they bring together modules in the way your site uses them. Data in Hiera can then customize modules or roles in node-specific ways.

For example puppetlabs-nginx (and jfryman's nginx module that it's based on) configures a yum repo hosted on nginx.org, and they don't allow you to specify custom templates for configuration files. That means I would have to fork and customize the puppet module if I wanted to use my own yum repo with a version of nginx compiled with ngx_pagespeed. It shouldn't be that way. I forked puppetlabs-nginx and added parameters to use custom templates, I'm going to continue working on it to make it more generic while still being easy for the 90% use case. I would like Puppet to be as simple as Ansible for the easy cases, and still be efficient and powerful when you need to customize.

GoDaddy's crippled NetApp filers

I just finished helping my friend Adam Aragon with an issue on his AntiPretty [NSFW-ish] alternative modeling site. It's a WordPress site running on GoDaddy. He was getting an error every time he tried to upload new images. Directory permissions were fine, as was all the PHP and WordPress stuff. GoDaddy support couldn't find a problem. So I logged in with SSH and tried to 'touch' a new file in the directory.

$ touch /path/wp-content/uploads/2013/05/testfile1
touch: cannot touch `/path/wp-content/uploads/2013/05/testfile1': File too large

A sad day: 428 days of uptime lost

My Linux workstation had been up for 428 days without a reboot. No cheating, I don't think I even logged out of my X session in that time. But today is a sad day, I had to move my workstation to a new room.

I considered getting a UPS and trying George's Frogger move from Seinfeld. But it probably wouldn't have worked.

The upside is the reboot restarted all the memory leaks, so I'm not using 1GB of swap anymore!