No Man's Patience

How I very much want to like No Man's Sky, but it sure isn't making it easy!

Notes from a grumpy old ex-game coder.

I've probably put in a dozen hours or more playing NMS on the PS4 the last few days, and there's a lot about the game that I like, in terms of the atmosphere and just the general idea of procedural generation.

It's such a great pity that the flaws in the interface make it so hard to just sit back and flow through the game; you're always fighting it. After bashing my head against it for a while, I think the designers fell afoul a very common cognitive blind-spot -- they didn't spend enough time thinking about how players (especially casual, older players like myself) would actually play the game. And after all, if it was easy for them (the developers) to play, it should be easy for everyone else, right?

Wrong. We are at a huge experience and skills deficit. We don't have all the implicit knowledge built up as they developed the game. It really feels like they either didn't test with naive players, didn't test deep enough into the game, or didn't want to hear the results.

Here's a short list of outrageously annoying and really simple-to-fix problems with the game and interface.

• The cursor is often a single pixel and invisible. There's often also UI icons near the center of the screen that distract you and give you a false impression of where the aimpoint is.
• When you turn in space and stop, you keep turning a bit. This means it's damn near impossible to put the aimpoint where you want it, which makes space combat much more difficult. And BTW, there's no learning curve on space combat, you're almost always up against multiple enemies who will quickly slaughter you. The "sensitivity" controls are simplistic and don't do much.
• When you want to find something specific, you just have to wander around until you stumble upon it. You can scan when you're on the ground, but that just tells you the general class of a target. If you're looking for Copper and don't know it's the big floaty chunks, you will wander for hours. I know I did.
• Often the major differentiation between two types of minerals is color. I can imagine that color-blind players are not amused one whit.
• The only way to tell for sure what an asteroid or monolith is made of is to mine it. I've got a warp-capable starship that can't tell the difference between Iron and Copper.
• This wouldn't be so bad if the Galactic Market was well-stocked. But it isn't; the chances it will have what you need in the quantities you need it, esp. for raw materials, is just about zero.
• Only raw materials stack, which is not pleasant in a game with limited inventory. I've ended up spending almost all my money on Exosuit upgrades.
• On the subject of the market interface, your options are two; sell all units of a commodity, or scroll the number down to sell partial -- both of which are things a player will rarely want to do. What they want to do, in a game where inventory slots matter, is sell one full stack at a time or the partial stack.
• While we're bitching about stacks, when you mine, you get shown what you are mining and how much you've mined since you started mining (and you can only mine 10-15 seconds at a time). What you really want to know is how full is your current partial stack of that commodity!

I could go on, but I feel a need to save some things up for future cathartic posts...

Intelligent US Address formatting

I recently needed to have a good solution for parsing US postal addresses into a standard format. After a bit of quick googling I found a neat github project that does most of what I needed. A big shout-out to the Atlanta Journal Constitition newspaper for doing this.

If you're running python, all you have to do is 'pip install usaddress' and you're up and running.

I then wrote a quick python script to convert usaddress output to a standard format, like this:

[Optional Name,  though you really shouldn't pass it]
Address line 1
[Optional Address Line 2]
City
State
Zipcode
#

Here's the usaddr.py script (python 3.x):

#!/usr/bin/python

import sys
import usaddress

for addr in sys.argv[1:]:
  try:
    c = usaddress.tag(addr)[0]
    l = []
    
    breakfields = [
      'Recipient',
      'BuildingName',
      'LandmarkName',
      'CornerOf',
      'USPSBoxType',
      'AddressNumber',
      'AddressNumberPrefix',
      'PlaceName',
      'StateName',
      'ZipCode'
    ]
    
    multikill = {
      'AddressNumberPrefix' : [ 'AddressNumber' ]
    }

    for k,v in c.items():
      if k in breakfields:
        breakfields.remove(k)
        if k in multikill:
          for mk in multikill[k]:
            if mk in breakfields:
              breakfields.remove(mk)
        l = [i for i in l if i]
        if l != []:
          print(' '.join(l))
        l = [v]
      else:
        l.append(v)
        
    l = [i for i in l if i]
    if l != []:
      print(' '.join(l))
    print('#')
          
  except usaddress.RepeatedLabelError as e :
    print('Error: cannot parse address')

Since I'm running a LAMP stack on one of my servers that happens to have python also installed, I wrote a quick usaddr.php script to implement an API:

< ?php

foreach (explode('&', $_SERVER['QUERY_STRING']) as $chunk) {

  $param = explode("=", $chunk);

  if ($param) {

    $cmd = urldecode($param[0]);

    $arg = urldecode($param[1]);

        

    if ($cmd === 'addr') {

      $command = 'python /path/to/your/script/usaddr.py ' . escapeshellarg($arg);

      $output = shell_exec($command);

      echo $output;

    }

  }

}

?>

So this meant I could access http://myserver.com/myapi/usaddr.php?addr=[address to reformat] from other apps (like FileMaker) to do address reformatting.

Setting up Mail Relay behind NAT for local network on OS X

I recently had a need to set up an outbound mail relay on my home network so local machines behind the NAT, (ie: machines with 192.168.x.x addresses) could send an occasional status email.

It turns out that OSX has a built-in postfix mail server, and it's pretty easy to turn it on. Here are some step-by-step instructions.

1) Create a domain alias that points to the external IP address of your network (ie: what http://whatismyipaddress.com/ says is your IP address). If this address is static and unchanging, then you can just set up a CNAME in the DNS of some domain you own. If it tends to change because your ISP reallocates them from time to time, then use a service like http://dyn.com/ to maintain a dynamic dns listing. Many routers support dyn address updating, and there are also apps that will run on an internal machine to do it.

2) For any domain that you want to send mail for, make sure this new dns entry is listed in the SPF record in the DNS. See http://www.openspf.org/ for more information. This will help ensure that other mailservers will consider the mail to be legitimate.

3) On the machine that will be doing the forwarding, you now need to edit a few system files. You'll need a text editor for this -- if you've done any mucking around in the shell, you'll know how to do this.

/System/Library/LaunchDaemons/org.postfix.master.plist

This config file controls how postfix is launched. Here's a simple one that should work with OSX 10.5 or later. If the AbandonProcessGroup key does not appear in the original .plist file, you can delete those two lines (17-18):

/private/etc/postfix/main.cf

This config file describes how postfix does its business. We need to lock it down so it will only accept mail from machines in our local network. It's a long file, but there are two configuration lines you will need to change:

inet_interfaces = all

This tells the server to listen on all the network interfaces, not just the internal "loopback" one.

127.0.0.1/32 is the default "localhost" address for your current machine. If you only want to relay mail for your current machine (the one postfix is running on), you set mynetworks to be just that:

mynetworks = 127.0.0.1/32

If you also want to relay mail for other machines in your local network, you need to know what the network range is. Typically that is something like 192.168.1.xxx, in which case you'd add 192.168.1.0/24 as the network range (the 24 is the network mask, it says how many bits out of the 32 bits are fixed). So for example:

mynetworks = 127.0.0.1/32, 192.168.1.0/24

Finally, you need to restart postfix and let the system know about the changed configurations. In Terminal, do this:

sudo launchctl stop org.postfix.master

sudo launchctl start org.postfix.master
sudo postfix start

Test that it's working by trying to connect to the mailserver:

telnet 127.0.0.1 25

You should see something like this:

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

220 Trebor.local ESMTP Postfix

Now test your mail sending by, for example, trying to send mail to a gmail account. Hopefully you'll be good to go.

Conway's Game of Life in the Nand2Tetris Hack Machine

Nand2Tetris is a very interesting course that steps you all the way from simple logic gates up to an operating system. One of my sons is taking the course this semester, so I decided to shadow the assignments and ended up having a lot of fun.

I've posted a video on Vimeo of an implementation of Conway's Life for the Hack machine.

You can find the source code here. Enjoy!

Mac OS / Linux : Finding all the files that meet some criteria and returning an escaped list of the paths, sorted by length

This is another post "for the next poor bastard"

One of the side effects of running AnimEigo is that I end up with a lot of video on hard disc -- terabytes and terabytes of it. This is amusing when you consider my first hard drive cost $5000, stored 20MB, and shook the table when the disc heads moved.

When projects are finished, we archive all the video and project files just in case we need them later, and rather than just copying the files onto an archive disc (easy and fast), since much of the original source materials are uncompressed audio and video, compressing these files before archiving means we can save a few bucks and free up some hard drives for reuse and replicate the files on multiple drives as insurance.

The Mac finder has a built-in feature that lets you compress files and folders, but it occurred to me recently to check to see if there were better options.

After doing a little research, I settled on using pbzip2, the multicore implementation of bzip2, which seems to do a good job of compressing uncompressed video files -- often down to 20-25% of the original size. If you're using a Mac, the easiest way to install it is by using the fink package manager.

As pbzip2 is a command-line tool, you invoke it using the Terminal app, by typing something like this:

pbzip2 -v "path to the first file you want to compress" "path to the next file" ...

and pbzip happily goes off and (slowly) compresses the file for you. All fine and good, and you can just type the pbzip2 -v part and then drag files in from a folder window to enter the paths.

However, because I'm lazy and thus willing to spend many hours automating things to save myself a few seconds of drudgery, I started playing around in the default Bash shell that Terminal provides; it had been a while since I'd done more than trivial things in it and a refresher couldn't hurt.

The basic philosophy of Unix shell tools is "lots of little tools that do a small number of things well that you can hook up to do something complicated". You send the output of one tool into the next tool using a pipe, represented by the | character. Here's the command sequence I came up with:

find . \( -iname '*.aiff' -or -iname '*.aif' -or -iname '*.wav' -or \( -size +500000 -iname '*.mov' -not -iname '*ProRes*' -not -iname '*H264*' \) \) -print0 2> /dev/null | xargs -0 du -s | sort -n | cut -f 2 | while read line; do printf "%q " "$line" ; done ; echo

Here's what it does. The first part invokes the find command; this finds any file that ends in .aiff, .aif, .wav or .mov, with the added restriction that .mov files need to be at least 5GB long and have a name that doesn't include the strings ProRes or H264; this eliminates most if not all of the compressed video files. The -print0 command says to separate the output file paths with a nul character instead of a linefeed (needed so the next tool doesn't get confused by spaces in filenames), and the 2> /dev/null redirects any error messages to the great bit bucket in the sky.

Each path gets processed by xargs, which is a tool that lets you run other tools on each line. The -0 means use nul as the line delimiter, and it runs du -s (disk usage) on the file paths.

The output of that is a set of lines, each containing the length of the file in disk blocks plus the path, separated by a tab. This gets piped into the sort tool, which is told to sort the lines by their numeric value by using the -n flag; I want them in this order so pbzip2 can compress the smallest files first, freeing up space for the larger ones; often an archive drive will be almost full when I start to compress it.

Next the cut tool is used to extract the second field, which gets us back our list of paths, now sorted smallest to largest.

Finally, I need to put all these paths on a single line, separated by spaces, and properly escaped (spaces changed to "\ ", for example). There is a printf (print formatted) tool for this, but the "%q" formatting code that does the escaping is not implemented in the MacOS version of printf (bitch moan bitch moan). However, printf is also implemented as a built-in command in the Bash shell, and that version does implement "%q", so a little inline shell script will do what I need - it reads each line, prints it out escaped with a space after it, and then echos a blank line. The final result is a single long line containing all the file paths, which admittedly looks like crap but I can just copy it, type in pbzip2 -v (or any other compression command) and paste it in. Actually, given how pbzip2 spawns multiple threads and can chew up a lot of your cpu resources, you probably want to do something like nice -5 pbzip2 -v to make it a bit more polite.

This won't work if the filename has really weird characters in it, like carriage returns, but that isn't a problem for me.

Let me end with a big shoutout to all the contributors to the many postings on stackoverflow that helped me find the right tools and combinations.

PS: I later stumbled upon this excellent comparison of various compression tools which includes an efficiency/time tradeoff chart. Of course, depending on what you are compressing, your mileage may vary!

The anagram of Samarang is Anagrams

A few days ago I randomly stumbled upon the "check if two strings are anagrams" interview question, and it piqued my interest.

While it is immediately obvious there is an O(n-logn) algorithm for this -- sort the character arrays and compare them -- after a few moments I flashed on the O(n) algorithm for comparing two strings and determining if they are anagrams of each other.

This got me thinking, and I remembered the fun I had about 15 years ago assembling a huge list of english words and checking to see if the .com domain names were available. A little quick googling found some interesting word lists and I was off to the races.

I ended up implementing the O(n) comparison algorithm along with a data structure that chops up the word list by word length and then hashes the words using a hashing algorithm that causes potential anagrams to hash to the same bucket. However, checking all the words in each hash bucket is O(n^2)

Thus, it is typically much slower than the simpler nlogn algorithm in Python, because the lengths of the strings and the typical hash bucket are small, the python list.sort() function is highly optimized, and once you do have the strings and bucket sorted, the search for anagrams is O(n).

And O(n^2-logn) < O(n^2-n)

The moral of this story is, in order to figure out the best algorithm, you need to consider the whole problem. I fell in love with the O(n) comparison, but didn't notice for a while that changing the problem from "check if two strings are anagrams" to "find all the anagrams in a word list" made a big difference.

Still, it was a fun evening of coding, and my python is getting marginally less horrible.

You can find the python code, word lists and sample output here.

PS: Samarang is a place in Indonesia

Troubleshooting a Brother DCPL2540DW Wifi Configuration

This is one of those posts I'm mostly doing for the benefit of "The Next Poor Bastard". My Mom got a new DCPL2540DW printer the other day and it took a little headbanging to get it talking to the other devices in the house.

It's a fairly nice printer: built in scanner/copier, two-sided printing, WiFi with autoconfiguration, etc. Initial setup was easy, including one-touch wifi configuration.

Just one problem: you could see the printer on the network, but attempts to print to it just died -- nothing could talk to it. Connecting a machine to it via USB worked fine, so there was a problem with the configuration.

Condensing down about two hours of troubleshooting:

• The Wifi status printout page doesn't tell you anything useful, but the Network menus have a status option that does -- stuff like IP address.
• The Printer was connecting to the WiFi router fine, but instead of getting an address in the 192.168.2.X range like all the other devices, it was showing a 169.X.X.X address -- a self-assigned IP address.
• My guess is that the WiFi router doesn't like to route packets to 169.X.X.X addresses, but probably accepts packets from them. So all the local devices could see the printer's "hey guys, open for business" announcement, but not do anything with it.
• OK, several possible ways to fix this: tweak the router so it routes between 192.168.X.X and 169.X.X.X, force the printer to get its IP from the router, or the brute-force solution: give the printer a fixed IP address in the 192.168.2.X range.
• So brute-force it is: go into the router configuration DHCP settings, and adjust the range of IP addresses it gives out to connected devices. It was 192.168.2.2-254, I changed it to .2-249. Then go into the printer's settings, and there's an option for setting the IP address -- set it to 192.168.2.250.

Bing! Printing now works from everywhere; Macs, iPads (Airprint), even the phones.

Some other minor tidbits:

• The Mac drivers package only installs on Mac OS 10.7 and higher, but Mom had an older 10.6.8 machine. However, you can download individual components (CUPS printer driver and Twain scanner driver) and they do install.
• With the scanner driver installed, you can use Image Capture to control the printer and scan, even from a 10.6.8 machine.
• You can print from a 10.5.X machine using the generic PCL 6 CUPS driver, but not over the network; you have to use the USB connection.

I'm probably going to try and get remote email printing working at some point, if so I'll update this post.

Brother Support Pages for Printer