05 January 2022
Wordle is a simple online word game.
It is played by similar rules to the classic game
Mastermind, which is
known in the United States through the version licensed to Hasbro. Wordle is
played with words instead of arbitrary configurations of pegs or numbers, and
there is one added twist: each of your guesses must also be a word in
Wordle’s dictionary, not any arbitrary configuration of letters.
For those unfamiliar with Mastermind, I recommend clicking through to try the
game or reading this brief description of the rules: every day, the game
randomly chooses a five letter word from a predetermined list of possible
words. On each of your turns, you will guess a word that must be in a
dictionary contained by the game. If you guess the word within 6 turns, you
win. The game tracks your streak of wins and the number of guesses it takes
you to solve each one.
For each guess, the game will tell you whether each letter
- does not exist in the correct answer at all
- exists in the correct answer, but is at the incorrect location
- is the correct letter and is at the correct location
Each letter in the guess has a 1-1 correspondence with the same letter in
the answer, if there is one. This means that if the correct answer is “means”,
and you guess “green”, you would see the following:
The yellow color indicates that the chosen letter appears in the solution, but
that it is in the wrong place in our guess. The second “e” appears in grey,
because there is no second “e” in the solution. There must be a 1-1
correspondence between each guessed letter and a solution letter, and if they
were both highlighted in yellow, the “e” in the solution would correspond to
two letters in the guess.
Likewise, a green letter, indicating a letter in the correct location, takes
precedence over a letter in the incorrect location. So if the solution was
“rules”, we would see the following:
Here as before only one “e” is highlighted, but the second is in green rather
than the first being in yellow, since the letter in a correct location takes
precedence over a letter in an incorrect location.
I’d like to solve the puzzle
When I come across a puzzle like this, I’m immediately compelled to think about
optimal ways to play it. For example, the player is in the same knowledge
position at the beginning of the game every time. Since the opponent is not
adversarial (a random 5 letter word is picked, not a word intended to be
problematic for any particular guessing strategy), the optimal strategy
necessarily means guessing the same word first every time. Few human players
likely play this way.
In fact, given that the solution word is drawn from an unchanging list of words
before the game begins, this means you can just pre-calculate the optimal guess
given each previous guess and response from the game, laying out every possible
game state in a tree-shaped diagram.
I have written a solver for Wordle in Python which searches for an optimal (per
several rules, see below) game tree, and saves it in a JSON file, which turns
out to only contain about 15 KB when compressed. The optimal game tree for
Wordle therefore turns out to be pretty small!
How the solver works
The ideal game tree would have 3 constraints:
- Every guess is a word that the game will accept
- The maximum path to a solution is less than seven guesses
- The average path to a solution is minimized
I have made several simplifications in order to quickly get a reasonable
- Every guess is a word that the game will consider using as a solution
- Each guess results in the smallest number of live possible solutions, on
A constraint to keep the maximum path under seven guesses turns out not to be
necessary, because a solver with just the two constraints above will never take
more than five.
As it turns out, the game actually has two word lists: one is the set of words
it will consider using for solutions, just over 2000 words. The other is the
full dictionary of words it will accept as guesses. I found both these lists
guessable words would in same cases allow faster solves, but my solver is so
efficient even without this that I haven’t seen fit to implement it yet. So my
program only guesses words that could, in theory, be used by the program as
Likewise, always guessing the word that will result in (on average) the
smallest number of possible solutions is only an approximation of optimal
guessing. There are 2314^6 = 153525361154699100736 different possible routes
through the game, and so brute forcing your way to an optimal solution is
(while not unthinkable as it is in chess), probably unworkable in Python.
Clever pruning of the search tree should help (and I’m going to look at this at
some point), but is not as easy as it is in games with an opponent. In chess,
you can simplify the tree by assuming that the opponent will always make the
move that is worst for you. In Wordle, because the solution is random, you
have to always consider all possibilities and try to find the guess that is
optimal on average.
Still, though, reducing the average number of live solutions as much as
possible is a very good approximation of ideal play. Consider the following
After the second guess, the computer makes the interesting choice “pygmy”.
This may seem counter-intuitive. Not only do we already have two vowels
confirmed to be in the word, “i” and “e”, the guess actually contains “y”
twice, which reduces the number of letters contained in the guess. As a matter
of fact however, this guess is rather astute. The computer will always find
the solution on the fourth guess depending on the outcome of this query.
Using ‘A’ to mean absent, ‘P’ to mean present, and ‘C’ to mean correct:
AAAAA → liver
AAAPA → timer
AAPAA → giver
AACAA → tiger
PAAAA → viper
CAAAA → piper
Being in a point in the game tree where there are six descendent guesses
means that there are six possible solutions. Clearly, optimal behavior would be
to always find the solution on the next turn. A brute force search would find
this solution, but so would the heuristic of eliminating as many possibilities
as possible. With “pygmy”, we always eliminate five possible solutions, the
best result possible in this case.
Using a heuristic like this is much faster than a brute force search, because
it generates a guess in each situation without needing any recursion at all.
This was just a quick little project for me, taking a few hours, so the code is
relatively unoptimized. Determining the best first guess requires searching
every combination of possible guess and possible solution, and this takes
several hours. The complete tree is generated in only a few minutes after that.
Because it is stored as JSON, a player for the game is included that doesn’t
have to do any searching - it simply reads the next guess out of the game tree.
The solution is found, on average, in 3.51 guesses. In the worst case scenario,
the solution is found is 5 guesses. The histogram of outcomes is as follows:
1: 1 (0.0%)
2: 65 (2.8%)
3: 1085 (46.9%)
4: 1074 (46.4%)
5: 90 (3.9%)
If anyone improves on my solution by utilizing the complete dictionary or
achieving a brute force solve of the game, I would be curious to hear how much
you manage to improve on these statistics.
Jan 11: I discovered that the program was treating all guesses that resulted in
the same average reduction to the live possibilities as equivalent. We can,
without violating the constraints given above, choose to prefer guesses that
are also possible solutions to the current puzzle. Adopting this change reduced
the average solution distance from 3.68 to 3.51 guesses.
All of my code for this project
is open sourced, under a GPL3 license, on Github. This includes:
- words.txt - a list of the words used in possible solutions on Wordle
- solver.py - uses words.txt to generate the solution (solution.json)
- solution.json - a pregenerated copy of the solution
- stats.py - analyzes solution.json to generate the stats given above
- player.py - an interactive prompt that uses the solution to play the game
- game.py - in case you want to get addicted to Wordle rather than play once a
day, this game will generate endless puzzles for you to solve
All the Python code should run on recent versions of Python 3, including the
Pypy implementation for additional speed.
20 June 2021
Code accompanying this blog post is available on Github.
My partner is a research scientist working in a lab group. Every year, members
of the lab are expected to go on long driving trips to collect specimens for
research. This labor is shared among the members of the lab (in practice, the
graduate student members of the lab). So even though one lab member’s work
might only focus on a single species, a research trip might collect many
different species, even if the lab member who requires them is not present on
In practice, the planned destinations are known at the beginning of the
academic quarter. The trips need to be planned out in such a way that effort
and route duplication is minimized, while each individual only goes on as many
trips as their workload will bear.
While not a professional programmer or computer scientist by any means, I’ve
been tinkering with software long enough that coming up with ways to solve
problems like this is not that hard for me, and so I’ve been asked on several
occasions now to solve the routing problem for them. Having done that, I
thought I’d document my process (using entirely open source software and data)
and provide a few hacked together scripts to simplify the process.
The core piece of the puzzle is Vroom, a software
project explicitly designed to solve the vehicle routing problem in an
efficient way. It’s designed to integrate with
OSRM, a routing engine built with OpenStreetMap
data. If you’re curious about the system, I recommend this underappreciated
talk about Vroom.
Dealing with constraints can be complicated. Fortunately, I never had to worry
about blackout dates, because the routes once calculated could begin whenever
the driver was ready to start. That said, Vroom has one significant limitation:
the only metric it allows you to optimize is total drive time. If there is 3
weeks worth of work to do, and it minimizes total drive time to send one
vehicle on a very long trip, Vroom will do
You can’t tell Vroom to, for example, minimize the length of the longest
trip, or to make all trips as close together in length as possible. I thought
at first that this would make Vroom unusable for my tasks (it certainly doesn’t
seem like it would work for hot pizza delivery, for example), but with a lot of
handheld tinkering you can get it to work using availability constraints on the
vehicles. I also didn’t find any software that would handle the problem better.
While OSRM does provide a very generous free API, it’s still too limited for
the kind of routing I needed to do. Vroom works by requesting a routing matrix
from OSRM that contains the travel times between every pair of points you
submit. Sometimes I would be given a thousand or more destinations. This meant
I had to run a routing server myself. Fortunately I have an always-on server
that was more than up to the task.
You can find the routing server at
osrm-backend. There’s a Docker
image, but as I prefer to run software in different containers I simply built
it myself. The map data I needed (the US West subregion) is available to
download from Geofabrik.
After pre-processing, the data for the western United States is “only” 14 GB.
Very clear instructions for setting up the software are available on the
Github. You’ll probably want to use the MLD pipeline as the pre-processing step
is much faster. The page recommends the CH pipeline specifically for large
distance matrices (exactly what we have), but as we’re only going to submit a
small number of requests to this server in its lifetime, we can afford for
them to be slightly slower.
Vroom has a documented JSON format that describes the available vehicles, the
destinations they need to reach, and other constraints. I wrote a script to
create the necessary input for Vroom based on data in a spreadsheet (exported
to a TSV file).
The format of this file is
Latitude <tab> Longitude <tab> Skills <tab> Time
Skills is probably the most interesting feature I made available. Sometimes one
scientist is particular good at identifying one species, and so the plan is to
make sure they’re in the vehicle for every destination that involves collecting
that species. In the spreadsheet, the scientists can put a number (or several
numbers separated by a comma) in the corresponding Skills cell. Each vehicle
has a corresponding list of numbers which identify the scientists in each
vehicle (a scientist can go on multiple trips and so be in multiple vehicles).
Vroom will use the skills feature to only send a vehicle to a destination if
there is a scientist in the vehicle who can identify the specimen there.
The Time field is occasionally useful too. It’s an estimation of how long the
vehicle will spend at the destination not moving (in seconds). In most cases,
the scientists I was routing for would simply park the car, clip a branch of a
tree or shrub, and get back in the car, so this was effectively zero. In some
cases, however, the GPS point could be a mile or more from the road, meaning
that a longer hike was in order.
The scripts that I’m making available are very much hacked together, not up to
my usual standard for code I make public. For example, there are important
constants in the
makejson.py file that have to be set or adjusted when the
script is run. I didn’t come up with a convenient way to use
We have the already-discussed list of VEHICLES:
VEHICLES = [[1,2],[3,4]]
This identifies two vehicles / trips, where two scientists (numbered 1 and 2)
are in the first, while two other scientists (numbered 3 and 4) are in the
We have lists of starting coordinates and ending coordinates for each vehicle.
These are in
[longitude, latitude] format because that’s what Vroom itself
STARTCOORDS = [[-118.44, 34.068], [-118.44, 34.068]]
ENDCOORDS = [[-118.44, 34.068], [-118.44, 34.068]]
Note how the starting and ending coordinates for the vehicles are all the same
in this case. That’s a common pattern (starting and ending at the university),
but you’re not locked into doing that by any means.
Last we have TIMEAVAIL:
TIMEAVAIL = [400000, 400000] # seconds
This is the messiest part. It’s the total drive time allowed for each car. In
theory, this would be perfectly known to the participants in advance (they
could allow more time for stops, breaks, sleeping, etc., as needed). In
practice, though, it’s a balancing act. The research trips have to be done,
so even if finishing all of them would take more time than the scientists
theoretically have, put together, that just means that they have to find more
time from somewhere. So in practice the only workable approach that I found was
setting this value unreasonably high for each car, then reducing it
selectively to get routes that looked reasonable. When you can’t reduce it
any more without the router failing, you have a roughly optimal solution.
Vroom will print a large JSON file with the results (and the routes themselves
from OSRM in
polyline format), and so I have another script that takes the
result and prints the result as a long list of coordinates for each vehicle,
and also exports the routes (as lists of coordinates) to text files, so that
they can be shown on a map. Theoretically I ought to create KMLs or something
more useful, but this turned out to be more complicated than I had hoped to do
in Python and so I ran out of time.
My repository contains a
run.sh file that shows how I run the program for my
specific setup. Obviously you’ll need all the pieces in place (Vroom, OSRM on
a server, etc) in place for this to work. Running the script, even for a large
list of destinations, takes only 3-5 seconds, so re-running it while changing
the parameters is no big deal.
The results are extremely nice looking:
Thanks to GPS Visualizer for making it easy
to plot these routes.
This solution shows three different routes, constrained to a limit of about a
week per trip. Several of the points have additional constraints, such that
only one scientist can collect the sample, and this has still allowed for a
nice solution. In many cases, it’s possible for human intuition to come up with
a map that looks a lot like this one, but getting the exact order of
coordinates right turns out to be crucial in saving hours of drive time.
At the end of the day, I suspect my exact approach won’t be completely workable
for most people. You’ll have slightly different constraints than me, a
different input format for your data, or some other problem. That’s why I’m
just sticking these scripts online instead of spending the time to make sure
they’re as functional and well-written as they could be. Hopefully, others will
be able to look to them for help in solving their own routing problems.
13 January 2021
I’ve been playing FEZ for the first time recently, and came across this room where some NPCs are constructing a QR code on the wall:
This seemed very likely to be a puzzle. It’s almost a complete QR (most of what’s missing is the top left positioning square), and anything under the scaffolding can be reconstructed, so it stands some chance of scanning after a few fixups.
I imported the screenshot in GIMP. I combined it with another screenshot of the same scene to remove the characters, leaving only the scaffolding. I used the Color Exchange tool (Colors > Map > Color Exchange) to replace the lighter purple color in the background with white. I then used the Threshold tool (Colors > Threshold) to make anything else in the image black.
Then, in another layer, I created a grid precisely aligned with the blocks in the image to help me see and fill in the missing portions. GIMP has a tool for that (Filters > Render > Pattern > Grid). I replaced the partial blocks hidden by scaffolding with completed blocks. This result looked like this (the grid is at 25% opacity):
Unfortunately the result still wouldn’t scan, even when I added the third position symbol back to the top left. Too much seemed to be missing, at least for the apps I tried. But since this seemed to be a puzzle that the developers intended you to solve, I assumed there was something more here, so I broke out the QR code spec.
The first thing to understand about QR codes is that they’re relatively simple ways of encoding binary data (ones and zeros) using black (1) and white (0) blocks. Certain parts of the image are only used to align scanners: they don’t contain any data. Other parts contain format information: this tells a scanner what kind of QR code it is and how to decode it. I was hoping this QR code would contain valid format information, as otherwise I’d have to test all the possibilities. Here’s an illustration of the different parts of a QR code. Image credit: Wikipedia.
From the size (25x25), we know this is a V2 QR code, which doesn’t have version information. The first question to answer was whether the QR code still contained valid Format Information. This information is error corrected and duplicated within the image. It appears in the QR code in the sections highlighted in blue:
One of the Format Information bit strings seems complete: it combines the top-right portion with the bottom-left portion. I’m working with the assumption (that I might need to change later) that anything not in the top left quadrant is “finished” by the NPCs, meaning it’s trustworthy. Hopefully this means I won’t need to apply error correction to the Format Information.
Information in QR codes is masked (XORed with a known pattern) to break up large chunks of white or black that give scanners difficulty. The Format Information uses the static mask of “101010000010010”. Wikipedia incorrectly says this mask is “101011001010101”. The data in the QR code will use a different mask, chosen dynamically by the encoder so as to minimize large white or black chunks.
Here is where I began to be surprised by how far I could go using only GIMP. I expected to be importing the data into Python relatively quickly, but being able to toggle masks on the QR code directly and observe the result visually turned out to be very handy.
Using the new “Exclusion” layer mode, you can selectively invert parts of the underlying image using a black and white layer. Take care to observe the correct bit order when creating the mask (MSB: for the Format Information, that means bottom to top, left to right), as seen here:
Applying the mask, we see the following:
Note that because of the way the GIMP Exclusion layer works, we have to use white as the “on” bit for the mask. Black would be more appropriate since that’s what the QR standard uses for an “on” bit.
According to the spec, the first two bits give the error correction level, and the third through fifth bits give the data mask information. That’s once again in MSB order, so bits 14 and 13, and 12, 11, 10, respectively. In this case, we have “11” for the error correction level, and “001” for the data mask. That tells us that we have error correction level “Q” (which we’re ignoring for now), and “i mod 2 = 0” for the mask (this is one of several options for a mask that is chosen by an encoder). This means that the mask is active (flipping the underlying data bits) for every even number row, starting at 0 for the top row.
We can quickly create such a mask in GIMP by making a small portion of it (just two blocks, in this case), then expanding that with Filters > Map > Tile. We then trim out any portion that is part of a position symbol, timing symbol, alignment symbol, quiet zone, or Format Information (so that only the data will be masked). The spec (and other references) are pretty clear about what these areas are, so I’ll skip to the result here:
We’re now ready to begin trying to read the raw data! The order of bits is detailed in section 8.7 of the spec. We start with the bottom right block, then go left, then go back and up, then left again, reading upwards in a step-wise pattern. When we reach the position symbol at the top we move to the 2 block wide column to the left, and this time step down. (Discovering that QR code data starts with the bottom right and that it ends with error correction data gave me hope that the QR code was decipherable.) The reading order for the intricate sections is given precisely in the spec. I’ve indicated the interesting parts below.
Section 8.4 of the spec indicates that the data stream is prefixed by a header, where the first four bits are a “mode indicator”, followed by a length indicator. If you’re just encoding raw binary data in a QR code, you may as well just use 8 bit bytes, which is mode “0100”. But if you’re representing numeric or alphanumeric data, you can encode it more efficiently. The FEZ QR code begins with “0010”, which indicates an alphanumeric QR code.
This means that the next 9 bits (for a V2 QR code) are used to give the number of characters in the data. The next 9 bits of our QR code are “000010111”, which is the number 23 represented in binary.
For maximum efficiency, alphanumeric mode encodes 45 possible characters into blocks of 2 characters per 11 bits. (45^2 / 2^11 ≈ 98.9%, which is pretty good efficiency). To access the characters, we divide the 11 bit integer by 45. The result without the remainder is the first character, and the remainder is the second character.
Each number represents a character according to its position in the following string:
Note that we have an odd number of characters, so to save every bit possible the last character will be encoded as a single 6 bit value, instead of wasting another 5 bits.
With all this information in hand, we can transcribe the data from the QR code. Anything after the last piece of data will be part of the error correction bits, which we’re hoping we don’t need. Now that we know how much data there is (22 × 11 / 2 + 6 bits), we can also identify each portion of the QR code by its function:
Blue represents Format Information (as before). Green represents data. Red represents error correction. Anything purely black or white is a static part of the QR code.
I have transcribed the data like so:
10011011100: 27 29 (R T)
11001101111: 36 27 (Space R)
10100111101: 29 36 (T Space)
01111001110: 21 29 (L T)
11001101111: 36 27 (Space R)
10100111101: 29 36 (T Space)
01111001110: 21 29 (L T)
11001101001: 36 21 (Space L)
10100111101: 29 36 (T Space)
01111001110: 21 29 (L T)
11001101111: 36 27 (Space R)
011101: 29 (T)
In other words, the data says “RT RT LT RT LT LT LT RT”. This is a sequence of inputs you can make with the controller in FEZ. You usually receive a collectable for doing so.
I also wrote some Python code to do the decoding automatically, which you can find here: fez_qr.py.
With all the information we now have, we can actually generate the final, corrected version of the QR code. To do that, we’ll need a library that allows us to set the parameters manually (because an automatic QR code creation tool might choose a different mask or error correction level, giving a completely different result). I used the one here. Exporting the resulting SVG and layering the result on the original, we can see that it’s nearly a perfect match!
Having finished all this, I reopened FEZ, and entered the input sequence. And NOTHING HAPPENED. I tried multiple times to make sure I wasn’t screwing anything up.
I looked up the puzzle online, and apparently a completed version of the QR code is in a later stage of the game, and this unfinished one is just for “lore”. I find this explanation frustrating, because it clearly seems to have been designed so that an enterprising person could reconstruct the code themselves. Everything you need is there. Yet I have to admit… it doesn’t work. You can’t enter the code in this room, you have to enter it elsewhere.
I think this is a mistake on the game designers’ part. If a puzzle gives you everything you need to solve it, the solution should work. It’s rather frustrating to put that much effort into solving a puzzle only to find out that your effort wasn’t anticipated, and the puzzle is actually solved for you later in the game.
16 September 2020
I’ve been watching a lot of early-2000s sci-fi TV recently, and I’ve noticed
that just about every show is plagued by a very specific problem.
Take Stargate SG-1 as a pretty typical example. According to an interview
with one of its producers,
the first two seasons of the show were shot on 16mm film, and then until season
eight the show was shot on 35mm film. It was finished along with the effects
compositing in standard definition, of course. For most if its runtime it was a
full screen show as well. All this, of course, means that the first seven
seasons of the show aren’t that great looking by today’s standards.
By 2004, however, audiences and networks wanted to see shows in HD, and this
meant a different set of expectations and trade-offs. Stargate SG-1 moved to
HD digital acquisition for its eight season. The results are, well…
The choice of this image is obviously selective. What’s happening here is that
the dynamic range of the scene (the difference in luminance between the darkest
part of the image and the brightest part) is too great to be captured by the
camera. The highlights are blown out. This scene is one of the worst looking
in the show, but as several other shots indicate, the same problems are obvious
in many other scenes too.
In addition to frequent blow-out issues, these HD shots suffer from quite a
bit of dynamic range compression and loss of detail in the highlights. This is
visible as a kind of “pastelization” of bright colors, making them “flatter”
looking than they ought to be, even though their saturation is at a normal or
even raised level.
While the increase in resolution is obviously appreciated, it also comes with
quite a bit of quality loss in terms of accurate, film-like colors. Hair and
faces tend to glow white or even bluish under any bright light, colors are harsh
and crushed, and many shots now have an ugly “plastic” look to them that further
exacerbates issues with the CGI and other effects shots. Even though many of
these shots wouldn’t look photorealistic without effects, the absence of
photorealistic images makes the effects shots feel worse on a subconscious
level, the plastic glow giving the show a queasy fever-dream veneer.
Primarily, these issues effect shots in uncontrolled outdoor lighting.
However, SG-1’s sister show, Stargate Atlantis, is frequently affected
by lighting issues even when shooting indoors.
I’m not sure what the issue here was. Perhaps it can be chalked up to different
crews and the producers having their primary focus on the last season of SG-1,
airing around the same time. However, I’d also point to the Atlantis crew’s
unwillingness to compromise on the shots. The SG-1 outtakes I’ve used as
examples are generally exceptions — what the show looks like when it looks
noticeably bad. The weird plastic-like appearance of HD remains on all the
shots, but they usually managed to avoid too many issues with the highlights.
I don’t have firm figures on how much the filming changed due to HD, but it’s
pretty indicative that I had to look through quite a few episodes of SG-1 before
I could find the kind of shot I was thinking about. Many episodes from the last
three seasons are shot entirely inside with carefully controlled lighting, on
some combination of the Stargate Command set and a few others. As I remember it,
this marked a clear shift from the kind of “new planet” episode openers many of
the earlier seasons depended on.
Even when shots in the later seasons of SG-1 are clearly outside, they tend to
be in places where the characters are in shadow, for example in the ubiquitous
forest scenes, inside a warehouse, or on the shadowed side of a building or
mountain. Anything to control the bright highlights!
In other scenes, heavy lighting was used to blend the scene together better and
give it a unified aesthetic, removing any harsh glows.
Quick trivia question: what do Attack of the Clones, Stargate SG-1, and
Battlestar Galactica all have in common? Answer: they were all shot on the
same camera, the Sony F900 HD series. This was an early professional HD camera
which was pretty readily available and affordable even for TV shows. It suffers
from a number of issues, including obviously the dynamic range issue, mentioned
above, but also the fact that anyone working with HD video this early was most
likely shooting in the same gamma (e.g. Rec. 709) as the final product, instead
of shooting with a log curve as
more modern productions would. Even if the image sensor in the F900 had been
fantastic, shooting this way would probably have crushed quite a bit of the
original detail out of the image, in a way that couldn’t then be recovered in
To be clear, the F900 wasn’t unique in having these problems, they were largely
a side-effect of being an early adopter of HD acquisition. That said, while many
episodes of SG-1 took care to avoid putting the worst aspects of HD video on
clear display, Attack of the Clones and Galactica didn’t work with these
limitations, and the results are quite frequently terrible.
The dated CGI of Attack is often credited for giving the film its plastic
look, and that’s certainly contributing given the number of shots that are
almost entirely greenscreened, but as you can see from examining these shots
closely, the lighting issues at play with this early HD camera is another
But now of course we come to Battlestar Galactica, which more than any other
show I can think of illustrates the pitfalls of shooting with HD video cameras.
Frankly, Battlestar Galactica is so bad it seems deliberate. Almost every
single shot in the show looks like this, it’s borderline unwatchable.
Unfortunately, it does in fact seem to be the case that some of this was
intended, as part of the creators playing with what to them was a new format.
An article on the use of HD
in Star Trek Enterprise and Battlestar Galactica describes the difficulties
that the creators of Galactica faced when they tried to achieve the right
aethestic for the show, which was “grungy”, “gritty”, with pushed grain. To
try to get this result on a TV shooting schedule, DP Steve McNutt developed
a realtime color management process for adjusting color on set while using
little, if any, post-production color correction.
As part of an attempt to replicate a “documentary-style”, shot-on-film look,
McNutt “[pushed] for video noise to emulate grain”, sometimes as high as +18dB.
The visual effects supervisor, Gary Hutzel, says that the DP “pushed gamma,
crushing the blacks, clipping the highlights”. That’s extremely telling. It’s
not worthwhile to try to parcel out blame, in hindsight, for a show shot almost
20 years ago; what’s important however is how the specific limitations of
in HD forced creators to experiment with novel techniques to try to get the
specific looks they wanted to achieve, with expectations often keyed to the
behavior and appearance of the film stock they were used to working with.
Even with the F900, Battlestar Galactica didn’t have to look as bad as it did,
but it’s easy to see how seeking the intensity and immediacy of a grainy film
stock led too many directors astray.
You might be thinking of the Battlestar Galactica mini-series that aired the
year before right now, and wondering why it looked so good. Indeed, it does
look fantastic. It was shot on 35mm film.
Now you might recall that the Battlestar Galactica miniseries is available on
Blu-ray. And it’s not upscaled — it looks great. So if everyone was
already shooting on 35mm film before HD video took hold, why not just
continue doing so when the networks wanted HD?
It turns out that most producers would have liked to stay on film, but
were forced to change due to budgetary demands. For example, SG-1 producer
John Lenic said (speaking about follow-up show Stargate Universe):
Film still is from a look perspective and in my opinion, the best looking
format to shoot on. … The only reason that we didn’t go film is financial.
Film still is roughly $18,000 more per episode than digital as you have all
the developing and transferring of the film to do after it is shot.
This transfer cost was something that producers were willing to bear when it
seemed to be necessary, but less so when HD promised a sharper image for less
money. In particular, continuing to use film when shooting for HD would have
meant having to do the transfers in HD as well, along with the same finishing
costs required for HD post-production, including the effects shots.
Additionally, the “cleaner” image of HD video, lacking in rough grain from film
stock, made effects compositing a bit easier to do in HD.
With Battlestar Galactica as well, director Michael Rymer initially opposed
shooting with HD cameras. Here too, the reasons were financial, as it wouldn’t
have been possible to produce the show at all if it were shot on film. In the
Rymer indicates that he noticed many of the issues that I have pointed out. In
particular, he says “the video was picking up the fluorescent bars of various
consoles on our [spaceship interior] set”, and calls daylight exteriors “less
than satisfactory” and “the worst environment for HD”. That said, Rymer did
eventually come around to the look of the show on HD, saying that the sensor
noise “approximated film grain nicely” and that he could ultimately achieve the
desired aesthetic for the show. On this point I have to disagree, but again,
we’re looking back with 20 years of hindsight on how badly early HD footage
and CGI work has aged.
As it turns out, a ton of
productions were shot with the F900 camera, and cameras like it, and the effect
on TV production quality was fairly devastating. This isn’t a critique of
shooting digitally in general, of course; the technology has vastly improved
over the course of the last decade. Some shows, like The Big Bang Theory,
are so reliant on internal sets and carefully controlled lighting that the
limitations of shooting HD don’t really show. Others, like The Office, make
use of the raw harsh lighting to their advantage, creating an alienating and
sterile environment that pushes the presence of the camera in your face,
achieving a documentary tone that in my opinion Galactica never did.
Once you decide to go with digital acquisition and approximate the final color
grade in-camera, there’s basically no hope of an improved release (for home
video or otherwise) years down the road. On the other hand, it’s worth pointing
out that early seasons of SG-1, because they were shot on film, could be
released in HD one day. The difficulty of course is to find someone willing to
put up the money to have the original negatives scanned, and possibly
recomposite the effects in HD. For very popular shows, like Star Trek: TNG,
this has actually been done, but there’s little hope of the same for less
popular shows like Stargate.
Here too, then, there are tradeoffs. While we do have HD versions of the later
seasons of SG-1 (though only on streaming services, not Blu-ray), we have to
deal with the permanent damage to the image quality done by the acquisition
method. On the other hand, the earlier seasons have the potential to see
beautiful high definition scans, but this will likely never happen because it’s
simply too expensive.
12 September 2020
I recently read the book Jesus, Interrupted by Bart Ehrman. It’s a very nice book. However, he has the following very strange thing to say about the method historians follow: “If historians can only establish what probably happened, and miracles by their definition are the least probable occurrences, then more or less by definition, historians cannot establish that miracles have ever probably happened.”
This is the sort of statement that causes me to scratch my head and re-read several times because I get the strong feeling that it has to be wrong. Ehrman claims that historians must accept methodological naturalism. In other words, a historian can never conclude on the basis of their evidence that a supernatural event occurred. They must always conclude (whatever their personal convictions) whatever is most likely, and what is most likely by definition cannot be a miracle.
While this requires some unpacking already, it seems to me to be based on seriously mistaken notions of both supernatural events and likelihood itself. Before getting into that I think it’s worth considering the implications of such a view.
If I read Ehrman correctly, should he miss the second coming, even extensive video evidence of Jesus Christ returning through the clouds to rule the nations would not convince him of its happening. A supernatural event of any kind is simply above the pay grade of historians to comment on. These events are purely the realm of faith. (One wonders if he would believe his own eyes.)
Not only is this odd, it’s already directly in conflict with Ehrman’s own claims about other supernatural events. For example, this excerpt I’m commenting on is from a book which argues “if the findings of historical criticism are right, then some kinds of theological claims are certainly to be judged as inadequate and wrong-headed. It would be impossible, I should think, to argue that the Bible is a unified whole, inerrant in all its parts, inspired by God in every way.”
Let’s consider what the evangelical claim that the Bible was verbally inspired by God entails, counting the instances of miraculous activity. Conservatives claim that each word of the Bible was inspired by God in the original autographs (1), the books were sufficiently preserved by scribal copying over time (2), and the correct books were canonized by the winning side in the theological battles of the early church (3). Ehrman is on the record as thinking these claims are untenable in the light of the historical evidence (e.g. in light of contradictions in the text), and in fact has dedicated this very book to proving that point. How is this possible if all claims of miracles are the proper domain of faith and not history?
In the book, Ehrman also pokes some well-deserved fun at the fundamentalist KJV-only movement. But what this sect claims, on any intelligible version of their views, is that God inspired the work of the translators. If this is a faith-claim, as it certainly must be, what are we to make of Ehrman and other historians hastening to point out that the translators of the KJV based their work on inadequate sources and made many mistakes? Why should that matter?
To try to clarify this, let me quote extensively what Ehrman has to say about miracles.
Miracles, by our very definition of the term, are virtually impossible events. Some people would say they are literally impossible, as violations of natural law: a person can’t walk on water any more than an iron bar can float on it. Other people would be a bit more accurate and say that there aren’t actually any laws in nature, written down somewhere, that can never be broken; but nature does work in highly predictable ways. That is what makes science possible. We would call a miracle an event that violates the way nature always, or almost always, works so as to make the event virtually, if not actually, impossible. The chances of a miracle occurring are infinitesimal. If that were not the case it would not be a miracle, just something weird that happened.
I think this quote points up the problem quite nicely. Ehrman is begging the question by slipping in the assumption of naturalism into what ought to be a defense of methodological naturalism. Ehrman’s definition of a miracle is quite acceptable, actually. I’ll quote it again: “[a miracle is] an event that violates the way nature always, or almost always, works…” His conclusion from this is that “The chances of a miracle occurring are infinitesimal.”
This simply does not follow. One cannot simply assume that “the way nature … works” is the same as what actually happens. Miracles are by definition supernatural occurrences! If you assume that the story of what happened at some point in the past can be filled out entirely by causal or quasi-causal chains in the natural order of the universe, you’ve simply assumed that miracles do not happen. That, of course, assumes far too much for Ehrman.
What Ehrman can safely conclude is that the chances of a miracle occurring naturally are infinitesimal. To get from there to the claim that historians are justified in rejecting these explanations in all cases requires much more work. I can think of several ways to do so.
For example, maybe Ehrman wants to position the historian as a sort of scientist, who can’t consider supernatural occurrences precisely because they’re supernatural — they’re in the wrong domain. On this view the job of the historian is to reconstruct the best possible naturalistic explanation for what happened in the past.
One problem with this interpretation is simply that Ehrman doesn’t seem to be saying this. He repeatedly asserts that the problem with miracles for the historian is their unlikelihood. He writes:
Historians can only establish what probably happened in the past. They cannot show that a miracle, the least likely occurrence, is the most likely occurrence.
The bigger problem with this view is that it’s just a silly and arbitrary restriction of the historian’s task. Surely the job of the historian is not like that of the scientist at all! The scientist must construct the most plausible account of the behavior of the natural world. The historian, on the other hand, is tasked with giving the most plausible account of what actually happened in the past. If the most plausible account involves some supernatural activity (recall the example I gave of video evidence), so be it.
I think the most likely explanation for Ehrman’s reticence to consider supernatural explanations is that they’re an enormous unknown. In order to get from the claim that miracles are extremely unlikely to happen naturally to the claim that they’re extremely unlikely, one has to have a prior for how likely non-natural events are. In other words, do miracles happen? It is not impossible to imagine a world in which miracles happen all the time; indeed, some evangelicals believe we live in such a world. Accounts of miraculous healings and near-death trips to heaven appear regularly in media targeted at conservative Christians. Ehrman (as an agnostic) does not take these accounts seriously. He (and I) don’t believe we live in a world where miracles are common. That said, there’s an enormous difference between the claim that miracles are (at least) uncommon, and the claim that a miracle is always “the least likely occurrence”.
This leaves a vast gray area. For Ehrman, we have no convincing evidence of miracles that would lead him to set a high enough prior for them to figure in many historical explanations. On the other hand, he is unwilling to take the materialistic stance that miracles are impossible and always ruled out as possible accounts of what really happened. Thus he tries to bracket off these concerns as completely as possible from the historian’s task.
If this is really the best reading of what Ehrman wants to do, it’s hard to have methodological objections. It would be an enormous issue for historians if at every turn they had to speculate endlessly about the appropriate prior likelihood of supernatural intervention in the normal course of the universe. Given that miracles are at least not every-day occurrences, historians seem to be justified in attempting to find straightforward scientific accounts of past events. I have no objection to this.
What I continue to object to is the specific defense offered by Ehrman. He writes about the resurrection of Jesus, as a purported historical miracle:
The resurrection is not least likely because of any anti-Christian bias. It is the least likely because people do not come back to life, never to die again, after they are well and truly dead.
In other words, he follows up his claim that his assignment of low probability to the resurrection is not the result of anti-Christian bias with what’s probably the only blatant expression of anti-Christian bias in the whole book! If it is simply a fact that people do not come back to life, then Jesus did not come back to life. No wonder the resurrection has such a low probability in his estimation!
What Ehrman presumably means is that under some unclear historian-centric notion of probability, the probability of the resurrection of Jesus is extraordinarily low. This is a notion of probability that makes naturalistic assumptions, not because the probability of supernatural events is low (which would beg the question), but because it is appropriate for historians to bracket off these types of considerations. In other words, it is because the likelihood of a supernatural explanation’s being true is indeterminable that historians must steadfastly refuse to speculate about them.
This passage remains remarkably unclear. If something like this is what Ehrman means, he would do well to say so directly. Meanwhile, those interested in more general answers to these questions must consider the matter more holistically. What one has to think about is quite simply the plausibility of two more or less complete descriptions of the entire world. Which makes more sense: the view that miracles sometimes occur and are the best explanations for some past events? Or that there have never been miracles and all past events are explained in a naturalistic fashion? This is a difficult question to answer — but it is not a question from which reason must be banished as a matter of faith.
After writing this, I subsequently read one of Ehrman’s other books, How Jesus Became God. In this book, written about five years after Jesus, Interrupted, Ehrman has a more nuanced take on the historian’s role. For example, Ehrman writes,
It is not appropriate for a historian to presuppose a perspective or worldview that is not generally held. “Historians” who try to explain the founding of the United States or the outcome of the First World War by invoking the visitation of Martians as a major factor of causality will not get a wide hearing from other historians—and will not, in fact, be considered to be engaging in serious historiography. Such a view presupposes notions that are not generally held—that there are advanced life-forms outside our experience, that some of them live on another planet within our solar system, that these other beings have sometimes visited the earth, and that their visitation is what determined the outcome of significant historical events.
This is a useful comment. Something about the historian’s task prevents them from invoking beings or phenomena that are not accepted already by a majority of other historians and scientists. In perhaps the best version of his view in the book, Ehrman continues:
The supernatural explanation, on the other hand, cannot be appealed to as a historical response because (1) historians have no access to the supernatural realm, and (2) it requires a set of theological beliefs that are not generally held by all historians doing this kind of investigation.
Here Ehrman drives the cleanest wedge between the question “what actually happened?”, and the historian’s question, which is (here) seemingly “what is the most plausible historical-naturalistic reconstruction of what happened?” This confirms that my previous suggestion that Ehrman wants to bracket off supernatural concerns from history may have some truth to it.
Unfortunately, Ehrman is not always so clear is this book. Later, he returns to making almost exactly the same claim that he made in Jesus, Interrupted:
But simply looking at the matter from a historical point of view, any of these views is more plausible than the claim that God raised Jesus physically from the dead. A resurrection would be a miracle and as such would defy all “probability.” Otherwise, it wouldn’t be a miracle. To say that an event that defies probability is more probable than something that is simply improbable is to fly in the face of anything that involves probability. Of course, it’s not likely that someone innocently moved the body, but there’s nothing inherently improbable about it.
Here Ehrman returns to the concern with probability and the strange claim that miracles are inherently the most improbable explanation. This suggests, contra the statements made earlier in the same chapter, that the historian is concerned with the question “what actually happened?” and is simply inferring to the most plausible explanation. As I argued above, anyone truly committed to answering this question cannot simply forswear any non-naturalistic explanations, because there’s simply no way to show a priori that the supernatural will never figure in the most reasonable account of historical events.