Last week TfL released some data about crowding on Tube trains. There’s a good write-up by Diamond Geezer.

Basically, TfL give each 15-minute slot on each line in each direction a rating from 1 (quiet) to 6 (stuffed). I thought I’d have a go at turning that data into some heatmaps, so you can pick out the busiest times more easily.

Now, this was my first time trying to make heatmaps, my first time using Seaborn and my first time using Pandas. The plots are not great. I basically got to the end of my tether trying to sort the code out and ended up spending a chunk of time polishing the resulting images. Even then…

Before we go any further, all the plots are linked to the SVG files if you want to have a closer look as the labels appear quite small. Also, I’m only going to describe the oddities in the plots, not draw conclusions from them — others can do that if they’re interested.

Let’s start with the Waterloo & City line because it’s simplest:

Tube crowding on the Waterloo & City line

Hopefully you can pick out the labels on the left showing the source and destination stations, the times (5am to 2am) and the change in colour showing how crowded the trains are. It’s quite easy to pick out the peaks here: Waterloo to Bank around 8 and 9am, Bank to Waterloo around 5 and 6pm.

Now, in alphabetical order, let’s go through the rest. Bakerloo:

Tube crowding on the Bakerloo line

Note here something that applies to the rest: not all pairs of stations are listed as I was having trouble making them legible. I tend to leave out every other pair so, above, it’s implied that after Harrow & Wealdstone → Kenton and before South Kenton → North Wembley there is Kenton → South Kenton. But, as we’ll see, this isn’t always the case.

Looking at the early morning services in Kenton and Wembley it is pretty obvious, as Diamond Geezer pointed out, that someone has been mucking about with the data.

Next, Central:

Tube crowding on the Central line

This was one of the first plots I “tidied” which is why the scale is missing. Note here the apparent disappearance of lots of travellers in the top-left and bottom-right of each plot. This is because of branching lines. I’ve tried to keep in the first station for each branch for each direction but it isn’t always the case. You may want to read these alongside the Tube map.

Circle:

Tube crowding on the Circle line

Not this is only the “Outer Rail” and “Inner Rail” — clockwise and anti-clockwise — sections of the Circle line, starting at Edgware Road. I left off the “jug handle” between Paddington and Hammersmith, which was pretty quiet anyway. In fact the whole line is fairly quiet — something DG pointed out — so the data is a bit suspect.

District:

Tube crowding on the District line

All those cut-off bits are because of the five branches in west London. It’s a weird line for historical reasons. Again I’m sceptical about the data because the District is the busiest sub-surface line, but the plots show that to be the Hammersmith & City:

Tube crowding on the Hammersmith & City line

No branches. Hurrah. Next.

Jubilee:

Tube crowding on the Jubilee line

I love how Canary Wharf slices across these charts.

Let’s sod off to Metro-land:

Tube crowding on the Metropolitan line

The Metropolitan line appears to be weirdly spacious in the evenings given how rammed it is in the morning. I’ve no idea if either is true to life.

More branching fun with the Northern line:

Tube crowding on the Northern line

I know, it’s weird, I should have done separate charts or something but TfL need to get on with it and split it up.

Piccadilly:

Tube crowding on the Piccadilly line

Strange pattern there, but it’s another strange line. Thankfully the last one is straightforward, the Victoria:

Tube crowding on the Victoria line

Doesn’t look too bad, that one.

Sorry the charts were so poor, do let me know what I should’ve done differently, or just whinge — I’m just glad this is over.

I think I might be done with SS-GB, having just watched the third episode. My problems are mainly with the plot, and I know, I know, it’s an adaptation of Len Deighton’s novel but crikey.

I’ve criticised the ludicrous notion of a warm Nazi-Soviet alliance before but Sunday’s episode was the one with Marx getting dug up so that bloody well needs to go on the list. The series is set in November 1941: two years after Hitler said: “Everything I undertake is directed against the Russians,” (Andrew Nagorksi, The Greatest Battle) and over a year after the Nazis began drawing up invasion plans of the Soviet Union. By early 1941 these included the “hunger plan” that proposed that German troops seize grain, with the anticipation that this would cause the starvation of tens of millions of Soviet citizens. This didn’t come to pass but in the weeks after the German invasion the Germans took three million Soviet prisoners and within nine months two million of them were dead.

But mostly it’s the idea of the collusion between the highest ranks of the German army in Britain with the resistance — who in the fiction are still fighting each other in northern Britain — to help the British king escape SS custody.

The logical grounding for this appears to be the idea that the German officer corps was dominated by aristocrats who where a bit put out by the brash Nazis. Far easier to do a deal with their upper-class counterparts in Britain — the leaders of the resistance.

This is pretty threadbare stuff. There was indeed some resistance to the Nazis in the German armed forces, but generally split into camps that opposed the Nazis generally and those whose main concern was that the Nazis would lead Germany into another war that it would lose. These latter misgivings were washed away by German military successes. In the hypothetical situation of SS-GB, where Germany has triumphed both on the continent and in Britain, it would be reasonable to think that the morale of German army officers would be high and fears of losing on the battlefield low.

In any case, supposing this German opposition faction, the likelihood of it finding a counterpart among the British ruling classes is highly questionable given their support for fascism, anti-semitism and fear of communism. After the publication of footage of Elizabeth Windsor giving a Nazi salute as a child, Max Hastings reviewed this history in the Daily Mail. The Mail of course knows all about support for fascism, what with its owner Viscount Rothermere declaring: “Hurrah for the Blackshirts!” in 1934 and all that.

The argument against this is to point to those in the ruling classes who did oppose Nazism, not least Winston Churchill. To counter this it is worth understanding the nature of the Nazi threat to their interests. Historian Adam Tooze notes:

The originality of National Socialism was that rather than meekly accepting a place for Germany within a global economic order dominated by the affluent English speaking countries, Hitler sought to mobilise the pent-up frustrations of his population to mount an epic challenge to this order.

This is confirmed in Churchill’s own words, outlining his opposition to Nazism in private to Soviet ambassador Ivan Maisky in 1938:

Today, the greatest menace to the British Empire is German Nazism, with its idea of Berlin’s global hegemony. That is why at the present time, I spare no effort in the struggle against Hitler.

Churchill’s own words and actions are hardly ones that support the idea of opposing the brutality of Nazism in principle. Churchill bears much responsibility for the Bengal famine and its five million deaths; then secretary of state for India Leopold Amery noted that, “on the subject of India, [Churchill] is not quite sane” and that Amery didn’t “see much difference between [Churchill’s] outlook and Hitler’s.” In Churchill’s secret evidence to the Peel Commission of 1937, he says that it was “not wrong” for the “higher grade race” (of Europeans) to slaughter Native Americans and Australian Aborigines.

Much more could be said, but the point is not to confuse practical anti-Nazism with principled anti-fascism. In a different situation — India, Kenya, Malaya, wherever else — the anti-Nazism of the white-supremacist colonial torturer looks distinctly weak. (And, in Vietnam and Indonesia, Britain in fact re-armed Japanese troops to kill liberation fighters struggling to prevent the return of colonial domination.)

I have a lingering doubt as to the worth of this post, attacking an adaptation of a work of fiction written closer to the war than to now. What irks me, I think, is that whatever the shortcomings of Deighton’s original novel, to reproduce them now in 2017 without alteration is deeply suspect. To smear the Soviet Union in such a way is unforgivable, especially at a time when European politicians are ratcheting up the hysteria against Russia in incredibly dangerous ways, and supporting a Ukrainian government that openly and officially celebrates Nazi collaborators who murdered Jews and Poles.

I wonder about a class motivation underlying these themes, doing down the Soviet Union (and socialism by association in the popular mind) while dressing up British and German elites’ resistance to Nazism. Again, it is fiction, but I worry what people might take away from a story that purports to be based on historical fact but which is so disconnected from it.

As a journalist and a socialist, one of my interests is looking into and understanding the beliefs that underpin journalistic output.

By far the best way of understanding the mass media in capitalist democracies is the Propaganda Model from Herman and Chomsky’s Manufacturing Consent. Its greatest strength is that:

We do not use any kind of “conspiracy” hypothesis to explain mass-media performance.

Crucially when we come to the role of individuals in this system:

Most biased choices in the media arise from the pre-selection of right-thinking people, internalized preconceptions, and the adaptation of personnel to the constraints of ownership, organization, market, and political power.

I think the idea of internalised preconceptions is powerful and portable. It is the idea that things “must” be as they are because “that’s the way the world works” or whatever phrase you want to use.

Switching from politics (of a sort), one thing that we see time and again is hostility to provision of cycling infrastructure because, obviously, mass unrestricted motoring is “the way the world works.” Any infringement on that — perhaps by redistributing road space — is in essence an affront to the natural order of things to benefit an out-group.

Again, this isn’t because the media are in hock to the Road Haulage Association or whoever else, but it is the natural conclusion in a society where transport policy has been focused on the motor vehicle for nearly a century.

An even more extreme case exists in the United States, where the motor vehicle has been so dominant that it’s not unknown for roads in built-up areas to lack pavements.

Which, finally, brings me to what sparked this post: a Los Angeles Times story about the health risks of living near LA’s many motorways. It’s actually a good story and pulls together all of the health and development stuff well.

However, it has a blind spot that reveals the internalised preconception that motor vehicle use must remain dominant. This paragraph sums up the avenues for change that the authors see:

Cities could re-zone areas near heavy traffic to exclude new residential development or change their general plans to prohibit such uses, planning experts say. Officials could adopt ordinances or moratoriums on new residential development. Or they could strengthen building standards — as they have for seismic reasons — forcing developers to design buildings in a way that reduces residents’ exposure to polluted air.

In short: stop building homes near motorways or build them better.

At no point is this flipped on its head to tackle the source: the motorway and the cars on it. There is no suggestion that that motorway is the thing that should be stopped, be narrowed or ripped out.

“10% of land currently zoned for residential construction” is “within 1,000 feet” of LA’s motorways so banning such construction would be problematic. But, again, the motorways are treated as unchangeable facts about our world that we must live with and cannot challenge — to the point that it is not even raised, just assumed.

Warning: The contents of this post may not be up to date!

Sorry, but if you’re searching for answers about why you can’t reliably set environment variables in OS X (particularly PATH), the information below may be out of date.

Apple has changed the system so many times in so many different ways it’s difficult to find anything on the internet about this that is correct and up-to-date, so I’ve placed this warning on the top of every post I’ve written about it.

As of 2017-03-02 it appears that the best collection of information is the osx-env-sync project on GitHub and its associated issues.

Be wary of anything that you find on StackOverflow: the answers there may well have been correct at the time but incorrect now.

For reference, my own posts on this problem (newest first) are:

Trying to set a consistent environment in OS X is the single most frustrating thing I’ve ever tried to do on a Mac.

I’ve written three posts (1, 2, 3) in the past few years trying to get to the bottom of this but still I have problems. (That’s why that giant disclaimer is on the top of this post!)

This bit me again the other day when using a Jupyter notebook to export from TextExpander, with my favourite UnicodeDecodeError. Here’s an example:

-----------------------------------------------------------------
UnicodeEncodeError              Traceback (most recent call last)
<ipython-input-8-c4b3c1d6f2e6> in <module>()
      1 with open('/Users/robjwells/Desktop/testfile', 'w') as f:
----> 2     f.write('☃')

UnicodeEncodeError: 'ascii' codec can't encode character
'\u2603' in position 0: ordinal not in range(128)

Now, this machine has been upgraded all the way from 10.7 and my understanding is that if you’ve a fresh install of a more recent version then you should be OK as far as the encoding-related environment variables (LANG, LC_CTYPE, LC_ALL). A recent Python enhancement proposal includes the paragraph:

On Apple platforms (including both Mac OS X and iOS), this [ensuring Python is set to use UTF-8] is straightforward, as Apple guarantees that these [Python start-up] operations will always use UTF-8 to do the conversion.

If you’ve ready my posts linked above, particularly Locale in OS X and Launch Agents, a good question arises: “I thought you’d figured this out!”

Well, yeah, me too! I have a script that runs at start-up designed to set my environment up. But it’s not reliable — I think because it may run after other start-up items (both Launch Agents & Daemons, and the Login Items listed in System Preferences). One of those is my Jupyter notebook server, hence the problem above.

Having the environment script run as the root user doesn’t solve the problem — in fact it means it has no effect at all.

(My way of checking whether a program has inherited my script-set environment variables is to define RJWENVSET as well.)

And then there’s the PATH, which I used to set in the same script using launchctl setenv like my other environment variables. Which doesn’t work. There’s a suggestion on a GitHub issue for a tool meant to resolve this problem that the following command could reliably set the path:

sudo launchctl config user path $PATH

But that doesn’t work for me. You can swap user for system but I’m wary of doing that.

This problem is ludicrously frustrating. Apple deprecated /etc/launchd.conf “for security considerations” but it means there’s no reliable way of setting variables that will be inherited by your programs, and so no way of having a consistent environment.

What a pain in the arse.

Right now, I’m doing the following:

  • I have my environment.sh file set LANG, LC_ALL, LC_CTYPE, PYTHONPATH, RJWENVSET when it is run at start-up by Lingon (for “all users”, though I’m not sure if this is any more effective than using “me” — I’m the only user on this machine).
  • My Lingon start-up items have environment variables and their PATH set explicitly.
  • I currently restart other start-up items that don’t pick up the variables set by environment.sh — but that doesn’t solve the PATH problem so you’ll want to set that explicitly.

But for Pete’s sake I wish I could just define variables in ~/.launchd.conf and be done with it.

Catching up on my RSS feeds, I’ve just been through Dr Drang’s posts from nearly a year ago about switching away from TextExpander (1, 2, 3, 4). I’m a bit behind.

And since everything on here is effectively hero-worship to Dr Drang, it’s time to catch up. Well, that, or it’s actually sensible to think about switching away from TextExpander; the last update to the non-subscription version (5.1.4) was on February 21 2016.

There isn’t anything in the subscription version for me. I barely use the features of TextExpander 5, rarely create new snippets, and generally am bad at using it.

A TextExpander reminder notification.

I get the snippet reminder notification all the time, to the point where it’s frustrating. I am not good at remembering snippet abbreviations. That’s not necessarily helped by TextExpander encouraging me to create snippets for things that I type frequently for a short amount of time, only to then not type the phrase and forget the snippet exists entirely. Then we’re back to the snippet reminder again the next time I do type the phrase.

TextExpander’s inline search.

For a while I was using the inline search, which is really good. And then I forgot the shortcut for that and would periodically open the menubar — where of course the shortcut for inline search is not listed — and get frustrated each time.

I am bad at using TextExpander and I feel bad.

A TextExpander suggestion notification.

Because of this, I didn’t really want to switch snippets to Keyboard Maestro or one of the TextExpander-alikes.

Instead I settled on using LaunchBar’s snippets, which you search in exactly the same way that you search the main LaunchBar catalogue. Setting a keyboard shortcut for snippets and enabling “sub-search only” means that they’re not cluttering up your usual results either: they’re available when I type ⌃⌥⌘Space but otherwise out the way.

It’s a similar advantage to when I moved my Safari bookmarklets to LaunchBar: if I forget the shortcut I can just type what I want.

A search for ‘star’ in LaunchBar’s snippets.

But so far this is just like TextExpander’s own inline search. And because snippets are part of LaunchBar there are some restrictions in using them with LaunchBar. Why bother?

Well, the overriding reason is still that TextExpander 5 is done. Sooner or later it’s going to stop working. It’s not a matter of whether I should switch but what to.

LaunchBar is a good fit because I use it everywhere, all the time. The snippets are just text files on disk, and so can be deleted or renamed directly from LaunchBar like any other file. They can be created too by sending items to the Add Snippet action.

Honestly I was a bit dismissive when snippets appeared in LaunchBar (“Why wouldn’t you just use TextExpander?”) but as a long-standing LaunchBar user they slot into my workflow seamlessly.

The placeholders are more limited than in TextExpander, which is fine, and snippets are plain-text only, which is again fine as I had very few script snippets and fewer that I used. Anything I keep around will find a new home in Keyboard Maestro or FastScripts.

As with Keyboard Maestro, the snippet dates are formatted with Unicode patterns rather than strftime — which is a bit uncomfortable for me but never mind.

I do have one major criticism of LaunchBar though. If you followed the link to its snippet documentation you’ll perhaps notice the old-style LaunchBar interface. Why?

Help not yet available

The Help for LaunchBar 6 is already in the works and will be available soon. In the meantime you might take a look at the Help of LaunchBar 5. Most of the information found there is still valid for LaunchBar 6.

LaunchBar 6 was released in June 2014. Maybe the docs will be in LaunchBar 7? In the meantime, time to actually read that copy of Take Control of LaunchBar.

Exporting your bookmarks from TextExpander

TextExpander doesn’t actually have an export option but it’s pretty easy to get your snippets out of the settings file, which for me was a single XML plist file Settings.textexpander.

As a warning, at the end of this I still had to do some manual work to tidy the snippets up, and none of the placeholders (where applicable) are converted. Excluding spelling correction snippets, I only have about 120 (many of which I deleted), but if you have many more this could be a problem.

But with that in mind hopefully it shows how simple it is to get at the actual data and reconfigure it into something useful for you. It’s broken up into parts and isn’t the most refined because it’s the result of mucking about in a Jupyter notebook until I got something that worked well enough.

First, the set-up (with the TextExpander settings file copied to my desktop):

import plistlib
import json

with open('/Users/robjwells/Desktop/Settings.textexpander',
          mode='rb') as plist_file:
    plist_data = plistlib.load(plist_file)
snippets = plist_data['snippetsTE2']

If we then pretty-print the first snippet you can get an idea of the format:

{'abbreviation': ';rjw',
 'abbreviationMode': 0,
 'creationDate': datetime.datetime(2014, 1, 13, 9, 33, 41),
 'label': '',
 'lastUsed': datetime.datetime(2017, 2, 28, 18, 56, 15),
 'modificationDate': datetime.datetime(2014, 1, 13, 9, 33, 47),
 'plainText': 'robjwells',
 'snippetType': 0,
 'useCount': 11,
 'uuidString': '87B39A64-B704-46CC-A82D-C3BB07A9C9B4'}

It’s fairly trivial to then pick out the fields you want:

export_snippets = [
    {
        'abbr': snip['abbreviation'],
        'label': snip['label'],
        'text': snip['plainText'],
        'type': snip['snippetType']
    }
    for snip in snippets
]

The type field represents whether the snippet is plain text (0), AppleScript (2) or (3) a “shell” script (or Python or whatever). I imagine that type 1 is rich text, but I don’t have any rich text snippets. We’re keeping it around for use later.

And that first snippet is now:

{'abbr': ';rjw', 'label': '', 'text': 'robjwells', 'type': 0}

At this point, before doing any further processing, I wanted to get rid of the spelling correction snippets. The ordering of my snippets meant that I could just find the “header” snippet and exclude everything that follows.

for idx, snip in enumerate(export_snippets):
    if snip['text'] == '\u00bb Auto Correct - Spelling':
        # Start of spelling correction snippets
        break
export_snippets = export_snippets[:idx]

For those that remain we now create a filename where the snippet text will be saved:

import string

def make_safe_filename(text):
    invalid_chars = set('/:\t\n\\;')
    text = text.replace('/', '-')
    return ''.join(char for char in text
                   if char not in invalid_chars).strip()

for snip in export_snippets:
    abbr = make_safe_filename(snip['abbr'])
    label = make_safe_filename(snip['label'])
    if not label and len(snip['text']) < 20:
        label = make_safe_filename(snip['text'])
    if label and abbr:
        name = f'{abbr} - {label}'
    elif abbr:
        name = abbr
    snip['file'] = name.strip()

The way I strip the abbreviation and label down is fairly arbitrary, but I exclude characters that may cause problems on the file system and also the semicolon — my no-longer-needed snippet prefix.

The code above is not what I actually used last night to name the snippets, but it could have saved me some work in Name Mangler. The important thing to note is that, in TextExpander, if a snippet doesn’t have a label the snippet content is shown instead. That’s what the length check is doing: if there’s no label and the snippet text is short, use the snippet text itself to name the file.

Now we finally deal with the snippet type. The “shell” snippets aren’t necessarily shell scripts so the “shell-rename” extension is to prompt me to change it to something more appropriate (.py, .sh).

snippet_types = {
    0: 'txt',
    2: 'applescript',
    3: 'shell-rename',
    }
for snip in export_snippets:
    snip['file'] = '.'.join(
        [snip['file'], snippet_types[snip['type']]])

Printing that first snippet again:

{'abbr': ';rjw', 'label': '', 'text': 'robjwells',
 'type': 0, 'file': 'rjw.txt'}

And lastly writing the snippets out:

from pathlib import Path

snippets_folder = Path('/Users/robjwells/Desktop/te-snippets/')
snippets_folder.mkdir(exist_ok=True)
for snip in export_snippets:
    file_path = snippets_folder.joinpath(snip['file'])
    file_path.write_text(snip['text'])

I then went through each snippet to clear out ones that wouldn’t work in LaunchBar (shell and AppleScript snippets), ones I don’t use, and ones that needed placeholders updating. I also renamed many snippets to tidy things up.

I have left the abbreviation for several snippets in the filename, so that any snippet-typing habits aren’t completely wasted, and trained LaunchBar for a couple of snippets where I don’t want an old abbreviation hanging off the front (for example, “iso” gets me “Current Date (ISO Format)”).