Joseph Turner's Blog

Inspired by a blog post I read via Hacker News, I thought I’d create a list of things I did and didn’t do in 2012. You know, for posterity.

Computing

Work

• Granola Enterprise. A lot changed in MiserWare’s flagship product Granola Enterprise in 2012. In February we debuted a new queue runner-driven data model for our backend, allowing us to do much more with the data coming from clients. In April, we rolled out what we intended to do with that: a new historical view of organizational energy consumption, complete with insights into areas of wasted energy and proactive tips for eliminating it.
  We also changed our preferred payment mode to a monthly subscription plan, allowing users to pay at the end of the month out of savings they have already earned, and only pay for the machines that were active during that interval. The end goal of this was to transition to a more organic model. We are still in the process of improving our funnel.
• FatBatt. With the core product for Granola Enterprise maturing, we began to look for new areas to direct our development efforts. For me, as I’m sure for many, battery life in my laptop has always been a thorn in my side.  I began to examine ways to improve battery life, during which I noticed how little information we are actually given about our battery life. As an example, how long was your last battery discharge? How long does it take to fully charge your battery? I couldn’t answer these questions, and I thought this would be a good place to begin to approach a solution.
  The result is FatBatt, a program intended to help you make intelligent decisions about your energy consumption on your mobile devices. It offers insight into your past discharges as well as providing statistically accurate estimates about the future given your current setup. This lets you make decisions like what the appropriate tradeoff for monitor brightness versus battery life is.

“Major” side projects

• Sprout! Someone introduced me to the pen-and-paper game Sprouts, and I thought it would be fun and illuminating to try and create a version of this game using browser technology. I made it as far as the one-computer interface before I put it aside. Never released outside of my own development environment.
• Brackcity The purchase of a ping-pong table for the office spurred a huge amount of ping-pong competition among me and my coworkers. I created an interface for doing algebraically-correct rankings based on historical gameplay. Needless to say, the main result has been a further intensification of the competitiveness. Never released outside of the office.
• Silvi A couple months ago I set out to scratch an itch of mine by creating a web service to better organize my thoughts. The result is Silvi, a way to create trees out of your information. The original intent was to develop a better way to collaborate on documentation, but using it myself along the way illuminated lots of other uses for it: organizing class notes for school, documenting a new programming language I was learning in a structured way, organizing documentation for products and services, and more. This is the furthest I have taken a personal side project, and I am very excited to see where it goes.

Languages, tools, frameworks

Python, Javascript, CoffeeScript, LESS, C , C++, PHP, SQL.

Node, Flask, Pywin32, py2exe, OpenCV, numpy, sqlalchemy, WMI, MailChimp API.

Books

I read about 25 books this year. As in 2011, my focus was on postmodern fiction, but this year I branched out and read some of my old flames and some nonfiction. Highlights:

• Thinking, Fast and Slow – Daniel Kahneman
• 1Q84 – Haruki Murakami
• Freedom – Jonathan Franzen
• Machine Learning – Tom Mitchell
• The Book of Laughter and Forgetting – Milan Kundera
• (Assorted) – Edward Tufte
• The Unbearable Lightness of Being – Milan Kundera

Climbing

The opening of Crimpers climbing gym in Christiansburg fueled a lot of my growth in climbing, both via meeting new climbers and improving my technique and fitness level. Both contributed to me sending my first (and second) V5 boulder problem this year.

I climbed at several new areas this year.

• Grayson Highlands in southern VA was largely developed by my friend Aaron Parlier, and as luck would have it, we bumped into him on our first trip down. He pointed out a bunch of the hotspots, and the rest has been tip-shredding history.
• Though I had been down there once before, the release of the guidebook for bouldering at Moore’s Wall in NC led to me spending several weekends down there. The style of bouldering, with big compression moves, fits me well and there are a number of projects I’m eager to send there.
• With neither guide nor guidebook, it was hard to find the highlights, but a recent trip to The Hill in Peaks of Otter, VA showed some really interesting problems on interesting (if not entirely clean) rock. I have high hopes for this area in 2013, particularly if I can get some guidance.

Personal

I traveled to (and had a booth at) CES in Las Vegas. My first taste of Sin City was surprisingly delicious: I ate at quite a few good restaurants and had a generally large time. I also visited San Jose and San Francisco, and spent some time in Santa Barbara implementing a case study. I also spent quite a bit of time in Chicago in the early part of the year visiting my (now ex-) girlfriend Natalia.

I ate at some great restaurants, including Picasso and Nob Hill in Las Vegas, Sutro’s in San Francisco, and (actually this was late December 2011) Next in Chicago. In the upcoming year, I’d really like to go to minibar by José Andrés in Washington, DC.

I sold my house in Christiansburg during a decidedly down market. The house had been on the market for over a year, and it was such a relief to sell it before the winter. As part of my exodus, I rid myself of a huge number of belongings, greatly simplifying my life.

I bought a truck, my first entirely autonomous car purchase. The process, going through a dealer, certainly has a greasy feel to it, but in the end I am happy with my purchase. I outfitted the truck with a very, very cheap camper shell and built a sleeping platform: it is ready to act as my home for camping and climbing trips.

2013 goals

I didn’t explicitly establish goals for myself for 2012, though my goals certainly included:

• Sell my house (check!)
• Travel more (FAIL)
• Climb more (check!)
• Work on side projects (check!)
• Get back into good health (check!)
• Create an organic software sales model for MiserWare (ongoing, not a win yet)

For 2013, I’ve decided to be more ambitious. My goals include:

• Squat 2x BW
• Deadlift 2x BW
• Clean and jerk 1.4x BW
• Snatch 1.1x BW
• Press BW
• Run a 5:50 mile
• Climb V6 outdoors
• No injuries
• Break even on a side project
• Make 10x investment in FatBatt
• Read 50+ books, including the entire Terry Pratchett oeuvre
• Write and record 5 songs I don’t hate

As a follow on to my previous post about the Rocktown guidebook, I’d like to let everyone know about two projects that they should help support originating within the Southeastern bouldering community. These guidebooks are created by members of the community, and supporting them by purchasing the books helps encourage the creation of more guides, so don’t just do it for you – do it for me, too.

The first is a bouldering guidebook for Moore’s Wall near Winston-Salem, NC created by (Dr.) Adam Sokolow. In the new preferred style, it features color pictures and drawn-on route descriptions. It also features GPS-accurate locations, thanks to some clever hackery by Dr. Sokolow, as well as a ton of interesting history of the area. After receiving the guide last weeked, I made it out to Moore’s and was blown away by the line quality and the unique features of the stone. I had been once before, but was underwhelmed; knowing the location, difficulty, and quality of the lines a priori made a huge difference. Check it out!

The second is a guide for one of my favorite new areas, Grayson Highlands. Over the last few years there has been an explosion of new problems at Grayson, largely as a result of the efforts of one of the guidebook authors (and friend of mine), Aaron Parlier, who has also put a lot of effort to secure access within the park. If you haven’t been to Grayson yet, you definitely need to check it out, though a lot of the areas are closed now due to snowfall. The guide is currently preorder, but I have seen a preprint and can say that it is definitely worth the money.

In my daily browsings of Hacker News I found a PyCon talk by Alexey Kachayev called Functional Programming with Python. As I’ve discussed previously (and despite Guido’s dislike of the style), this is a programming style I find very clean and readable. Check out the talk for an interesting overview of some of the techniques.

At MiserWare, we recently started work on a new project, FatBatt. The new software seeks to improve battery life for mobile users with laptops, tablets, and smartphones. We are about to enter closed alpha.

I will post more about the new project as things start coming together (and I start spending fewer long nights working on it!), but for now you can check out the interactive demo and blog for some more information.

Yesterday, I returned to my computer, left on to gather data over the course of several power management state changes, only to discover that Windows Update had automatically rebooted my machine to complete the installation of some critical, unnamed update. My data collection was truncated and needed to be restarted.

As every time this happens, I was infuriated, but I took a bit of time to think about it a little more carefully. I realized that fundamentally this is a disconnect that has arisen as a result of improved power management. In the simpler times of PCs-instead-of-laptops and poor system support for power management, an automatic update would most likely happen when nothing else was going on. The systems were mostly powered on, and updating in the middle of the night was no big deal.

Today, though, I (and probably most people) suspend my laptop whenever I am not using it by closing the lid. This forces the machine to update when I am actually using my computer, a time when an update and reboot is rarely convenient. As a result, I often postpone the updates. So when can they happen without the pesky user (me) interrupting them? Only when my computer is on but not actively being used, which is more or less by definition when I have some long-running automatic task going, such as my data collection yesterday.

What is the solution to this? How can Windows update itself without interrupting my tasks and sending me into fits of trichotillomania? The simplest answer is to change the default. Instead of automatically updating and rebooting, automatically update then notify the user that a reboot needs to happen in order to complete the update. The package management functionality in Windows is already plenty capable of this type of deferred installation. Unfortunately, this option is currently not only NOT the default as of Windows 7, but is not an option at all. Instead we much choose between inopportune reboots or user-initiated updating. Windows needs to catch up to a time when power management works and is embraced by many (most?) users.

Sean Kearney and Zak Roper have put together a new guide for the excellent southern sandstone at Rocktown in northern Georgia. I’ve already ordered my copy and am stoked for a trip back down south. I may, however, wait until it isn’t 1,000 degrees outside.

Check out the new guide here

For better or worse, our web development workflow begins on in-house servers that are the same software stack as our development webservers, particularly for new features that may change the data model. I’m currently working on a new feature for the new-and-improved Granola Enterprise that adds interesting and actionable aggregate data at the group and installation level, a perfect feature to work on in our cloistered environment. Each developer maintains their own Apache name-based virtual host to track their feature branch and any different data they need to track.

Today, I’m taking a cross-country flight to California to set up a case study of the new energy footprint generation capabilities of Granola Enterprise. The flight is long (>5 hours) and has the double advantage of both plenty of room (seat next to me is empty) and in-flight Internets, so I figured I’d get some work done. Getting to the development environment is a piece of cake: we have an Internet-facing ssh server. Setting things up so I can load my vhost in a browser is slightly more complicated, but is ultimately pretty easy using ssh port forwarding.

For a single-host Apache instance, it is really, really easy. Just ssh into your server and forward a local port to port 80 on the internal development machine. If your ssh server is ssh.example.com, your username is example, and your internal development machine is developmentmachine, you could forward local port 8800 like this:

ssh -L 8800:developmentmachine:80 example@ssh.example.com

To get to your webpage, then, just go to http://localhost:8800 in your browser. Simple. It can be even simpler if you forward local port 80 instead of a non-privileged port, but in that case you need to run the command as root (or with sudo).

With virtual hosts, it’s only a bit tricker. Name-based virtual hosts work by looking at the hostname in the HTTP headers, so that information must be right to wind up in the right place. The solution is to give your own machine the same name as your target vhost in your /etc/hosts file. Using the example above, you’d add this line:

127.0.0.1   developmentmachine

Now, instead of going to localhost in your browser, go to the normal name of your development vhost (http://developmentmachine:8800), and tada! you’re in. Bonus points: if you use port 80 (again as root) all your bookmarks work.

Now to do some real work instead of writing blog posts

Or, fun with vector geometry and state machines!

Once again, I’m toying around with HTML5 canvas. As part of the larger project, I needed a way to generate simplified paths based on mouse drags where ‘simplified paths’ means polyline paths with as few internal points as possible to accurately express the movement of the mouse. I did a bit of research and came up with a bunch of papers on the topic, both for purely polyline simplification as well as polycurve simplification, but I decided it would be more fun to roll my own.

So I cranked up some math rock, busted out the notebook, and dusted off the brain cells dedicated to vector geometry.  The breakthrough is realizing that the algorithm can run at a single point on a time. Here is the resulting algorithm in pseudocode:

1. Initialize TMIN, LMIN, line_list = [].
2. Save the mousedown point (s). Push s onto line_list.
3. Save the point after the first mousemove (m). Push m onto line_list.
4. For each new point, n (i.e. on mousemove event):
1. Let p = n – m.
2. If ||p|| < LMIN:
1. Pop the last element from line_list.
2. Push n onto line_list.
3. Return.
3. Else:
1. Let o = m - s.
2. Let θ = arccos(o . p / (||o|| * ||p||)).
3. If θ < TMIN:
1. Pop the last element from line_list.
2. Push n onto line_list.
4. Else:
1. Push n onto line_list.
2. Let s = m.
5. Let m = n.

At the end, line_list is the simplified list. Here is a diagram of the relevant variables:

In normal words, the algorithms only include points that are close enough to the line established by the previous two points, assuming that the mouse has gone far enough away from the last point. The nice thing about this method relative to some more complex ones (e.g., least-squares fit to the original line) is that the algorithm only needs to hold the points of the resulting line and the latest point.

So DOES it work? You can give it a go here. Fork the code (Coffeescript) here.

If you need to select several discontinuous items (and/or ranges) from a Python list, you can use the operator module’s itemgetter second-order function. In the realm of lists, it accepts arguments as either integers or slice objects and returns a function object that when called on a list returns the elements specified.

What? Like this:

>>> from operator import itemgetter
>>> get_items = itemgetter(1, 4, 6, slice(8, 12))
>>> get_items
<operator.itemgetter object at 0x02160D70>
>>> get_items(range(20))
(1, 4, 6, [8, 9, 10, 11])

I’ll leave it as an exercise to the reader to figure out how to flatten the resulting tuple. If it proves challenging, I’d suggest trying some or all of the 99 Prolog Problems (but a list ain’t one?), in Python of course

TL;DR: A change to the ‘import site’ mechanism between Python 2.6 and 2.7 can mean silent application exit for applications with embedded Python.

As you may know from some of my previous posts, we at MiserWare leverage an embedded Python interpreter in our Granola product, with success despite the pitfalls and sometimes cryptic (or missing) documentation. This week, I upgraded us from Python 2.6.4 to Python 2.7.2. Everything was going smoothly, until I began testing installation packages  for software with an embedded interpreter on our clean (i.e. non-development environment) test systems.

Since the packaging, in this case MSI files, hadn’t changed significantly for the upcoming release, and since my tests on my development machine of the software itself had gone smoothly, I expected the installation tests to be what they usually are: tedious, boring, and successful. I fired up the installer, went through the brief UI sequence, hit Finish and… the software didn’t start. Hmm. I cleaned the system completely, re-installed, same problem. I locate the file on disk, double click, same problem. I run the program from a command prompt, same problem.

At this point, I figure that the problem is with the packaging itself, so I double check my WiX files, but everything seems to be fine. By this point, it still hadn’t occurred to me that the Python upgrade could be the problem, so I drop in a few debugging MessageBox calls, rebuild, and re-install. After a couple rounds of that, I discover that the software is failing inside the Py_Initialize call. Hmm.

I fire up an old development environment that hadn’t upgraded to Python 2.7 yet and load the project in the Visual Studio debugger. Running the program causes program exit with return value 1. What? All of our return value error codes are negative numbers. Was the embedded Python library causing the application itself to exit?

It turns out, that is exactly what was happening. After some step-debugging through the disassembly of python27.dll with the Python source tree open in a separate console, I finally locate the source of the exit in Python/pythonrun.c:

static void
initsite(void)
{
    PyObject *m;
    m = PyImport_ImportModule("site");
    if (m == NULL) {
        PyErr_Print();
        Py_Finalize();
        exit(1);
    }
    else {
        Py_DECREF(m);
    }
}

Yikes! Previously (at least, in 2.6.4), ‘import site‘ failure would at worst print an error message to the console if there was an attached console… in the case of Granola, a Windows GUI application, there wasn’t, so all I had to work with was an application exit code being raised from within a linked library. If I was going to be diplomatic, I’d say that the decision to accept that patch was made with the interpreter executable and not embedding applications in mind. And it gets better.

Now armed with the location of the failure, it’s time to fix the application. It turns out that the call to initsite() occurs prior to the parsing of the environment variables.  What that means is that PYTHONPATH, PYTHONHOME, and other ways to tell the interpreter where to find modules don’t have an impact on the search for the site module, though that is not documented. [N.B., the reason it worked on my development machine was that it DOES honor a Windows registry setting that the Python installer creates, pointing to its own lib path.]

Without detailing the pain I went through to determine this, I’ll tell you the answer. You basically have one option (in Windows at least): you need to lay out your Python modules relative to your embedding application just as they appear in the normal Python installation tree, which is to say putting them all in Lib/ and DLLs/ folders at the same directory level as your application. You can also optionally call Py_SetPythonHome to specify a different directory under which to search for Lib/ and DLLs/. Oh, and just in case you go looking for it, that layout isn’t documented; you’ll only the preferred layout for Unix systems which won’t work in Windows.

What’s the takeaway? For me, it just means one thing: don’t expect consistency out of future versions of Python, at least with regard to embedding applications.