Adam Fontenot

How I Defeated Wordle with Python

05 January 2022

Wordle is a simple online word game. It is played by similar rules to the classic game Mastermind, which is known in the United States through the version licensed to Hasbro. Wordle is played with words instead of arbitrary configurations of pegs or numbers, and there is one added twist: each of your guesses must also be a word in Wordle’s dictionary, not any arbitrary configuration of letters.

For those unfamiliar with Mastermind, I recommend clicking through to try the game or reading this brief description of the rules: every day, the game randomly chooses a five letter word from a predetermined list of possible words. On each of your turns, you will guess a word that must be in a dictionary contained by the game. If you guess the word within 6 turns, you win. The game tracks your streak of wins and the number of guesses it takes you to solve each one.

For each guess, the game will tell you whether each letter

Each letter in the guess has a 1-1 correspondence with the same letter in the answer, if there is one. This means that if the correct answer is “means”, and you guess “green”, you would see the following:

the word "green" with the first "e" and the "n" highlighted in yellow

The yellow color indicates that the chosen letter appears in the solution, but that it is in the wrong place in our guess. The second “e” appears in grey, because there is no second “e” in the solution. There must be a 1-1 correspondence between each guessed letter and a solution letter, and if they were both highlighted in yellow, the “e” in the solution would correspond to two letters in the guess.

Likewise, a green letter, indicating a letter in the correct location, takes precedence over a letter in the incorrect location. So if the solution was “rules”, we would see the following:

the word "green" with the "r" highlighted in yellow, and the second "e" highlighted in green

Here as before only one “e” is highlighted, but the second is in green rather than the first being in yellow, since the letter in a correct location takes precedence over a letter in an incorrect location.

I’d like to solve the puzzle

When I come across a puzzle like this, I’m immediately compelled to think about optimal ways to play it. For example, the player is in the same knowledge position at the beginning of the game every time. Since the opponent is not adversarial (a random 5 letter word is picked, not a word intended to be problematic for any particular guessing strategy), the optimal strategy necessarily means guessing the same word first every time. Few human players likely play this way.

In fact, given that the solution word is drawn from an unchanging list of words before the game begins, this means you can just pre-calculate the optimal guess given each previous guess and response from the game, laying out every possible game state in a tree-shaped diagram.

I have written a solver for Wordle in Python which searches for an optimal (per several rules, see below) game tree, and saves it in a JSON file, which turns out to only contain about 15 KB when compressed. The optimal game tree for Wordle therefore turns out to be pretty small!

How the solver works

The ideal game tree would have 3 constraints:

I have made several simplifications in order to quickly get a reasonable solution:

A constraint to keep the maximum path under seven guesses turns out not to be necessary, because a solver with just the two constraints above will never take more than five.

As it turns out, the game actually has two word lists: one is the set of words it will consider using for solutions, just over 2000 words. The other is the full dictionary of words it will accept as guesses. I found both these lists in the Javascript source code of the game. Clever use of the full list of guessable words would in same cases allow faster solves, but my solver is so efficient even without this that I haven’t seen fit to implement it yet. So my program only guesses words that could, in theory, be used by the program as solutions.

Likewise, always guessing the word that will result in (on average) the smallest number of possible solutions is only an approximation of optimal guessing. There are 2314^6 = 153525361154699100736 different possible routes through the game, and so brute forcing your way to an optimal solution is (while not unthinkable as it is in chess), probably unworkable in Python. Clever pruning of the search tree should help (and I’m going to look at this at some point), but is not as easy as it is in games with an opponent. In chess, you can simplify the tree by assuming that the opponent will always make the move that is worst for you. In Wordle, because the solution is random, you have to always consider all possibilities and try to find the guess that is optimal on average.

Still, though, reducing the average number of live solutions as much as possible is a very good approximation of ideal play. Consider the following situation:

A series of Wordle guesses: "raise", "fiend", "pyymy", followed by the correct solution, "tiger"

After the second guess, the computer makes the interesting choice “pygmy”. This may seem counter-intuitive. Not only do we already have two vowels confirmed to be in the word, “i” and “e”, the guess actually contains “y” twice, which reduces the number of letters contained in the guess. As a matter of fact however, this guess is rather astute. The computer will always find the solution on the fourth guess depending on the outcome of this query.

Using ‘A’ to mean absent, ‘P’ to mean present, and ‘C’ to mean correct:

AAAAA → liver
AAAPA → timer
AAPAA → giver
AACAA → tiger
PAAAA → viper
CAAAA → piper

Being in a point in the game tree where there are six descendent guesses means that there are six possible solutions. Clearly, optimal behavior would be to always find the solution on the next turn. A brute force search would find this solution, but so would the heuristic of eliminating as many possibilities as possible. With “pygmy”, we always eliminate five possible solutions, the best result possible in this case.

Using a heuristic like this is much faster than a brute force search, because it generates a guess in each situation without needing any recursion at all.


This was just a quick little project for me, taking a few hours, so the code is relatively unoptimized. Determining the best first guess requires searching every combination of possible guess and possible solution, and this takes several hours. The complete tree is generated in only a few minutes after that. Because it is stored as JSON, a player for the game is included that doesn’t have to do any searching - it simply reads the next guess out of the game tree.


The solution is found, on average, in 3.51 guesses. In the worst case scenario, the solution is found is 5 guesses. The histogram of outcomes is as follows:

1: 1 (0.0%)
2: 65 (2.8%)
3: 1085 (46.9%)
4: 1074 (46.4%)
5: 90 (3.9%)

If anyone improves on my solution by utilizing the complete dictionary or achieving a brute force solve of the game, I would be curious to hear how much you manage to improve on these statistics.


Jan 11: I discovered that the program was treating all guesses that resulted in the same average reduction to the live possibilities as equivalent. We can, without violating the constraints given above, choose to prefer guesses that are also possible solutions to the current puzzle. Adopting this change reduced the average solution distance from 3.68 to 3.51 guesses.


All of my code for this project is open sourced, under a GPL3 license, on Github. This includes:

All the Python code should run on recent versions of Python 3, including the Pypy implementation for additional speed.

Solving the vehicle routing problem for research biologists

20 June 2021

Code accompanying this blog post is available on Github.

My partner is a research scientist working in a lab group. Every year, members of the lab are expected to go on long driving trips to collect specimens for research. This labor is shared among the members of the lab (in practice, the graduate student members of the lab). So even though one lab member’s work might only focus on a single species, a research trip might collect many different species, even if the lab member who requires them is not present on the trip.

In practice, the planned destinations are known at the beginning of the academic quarter. The trips need to be planned out in such a way that effort and route duplication is minimized, while each individual only goes on as many trips as their workload will bear.

While not a professional programmer or computer scientist by any means, I’ve been tinkering with software long enough that coming up with ways to solve problems like this is not that hard for me, and so I’ve been asked on several occasions now to solve the routing problem for them. Having done that, I thought I’d document my process (using entirely open source software and data) and provide a few hacked together scripts to simplify the process.

The core piece of the puzzle is Vroom, a software project explicitly designed to solve the vehicle routing problem in an efficient way. It’s designed to integrate with OSRM, a routing engine built with OpenStreetMap data. If you’re curious about the system, I recommend this underappreciated talk about Vroom.

Dealing with constraints can be complicated. Fortunately, I never had to worry about blackout dates, because the routes once calculated could begin whenever the driver was ready to start. That said, Vroom has one significant limitation: the only metric it allows you to optimize is total drive time. If there is 3 weeks worth of work to do, and it minimizes total drive time to send one vehicle on a very long trip, Vroom will do that. You can’t tell Vroom to, for example, minimize the length of the longest trip, or to make all trips as close together in length as possible. I thought at first that this would make Vroom unusable for my tasks (it certainly doesn’t seem like it would work for hot pizza delivery, for example), but with a lot of handheld tinkering you can get it to work using availability constraints on the vehicles. I also didn’t find any software that would handle the problem better.

While OSRM does provide a very generous free API, it’s still too limited for the kind of routing I needed to do. Vroom works by requesting a routing matrix from OSRM that contains the travel times between every pair of points you submit. Sometimes I would be given a thousand or more destinations. This meant I had to run a routing server myself. Fortunately I have an always-on server that was more than up to the task.

You can find the routing server at osrm-backend. There’s a Docker image, but as I prefer to run software in different containers I simply built it myself. The map data I needed (the US West subregion) is available to download from Geofabrik. After pre-processing, the data for the western United States is “only” 14 GB.

Very clear instructions for setting up the software are available on the project’s wiki page on Github. You’ll probably want to use the MLD pipeline as the pre-processing step is much faster. The page recommends the CH pipeline specifically for large distance matrices (exactly what we have), but as we’re only going to submit a small number of requests to this server in its lifetime, we can afford for them to be slightly slower.

Vroom has a documented JSON format that describes the available vehicles, the destinations they need to reach, and other constraints. I wrote a script to create the necessary input for Vroom based on data in a spreadsheet (exported to a TSV file).

The format of this file is

Latitude <tab> Longitude <tab> Skills <tab> Time

Skills is probably the most interesting feature I made available. Sometimes one scientist is particular good at identifying one species, and so the plan is to make sure they’re in the vehicle for every destination that involves collecting that species. In the spreadsheet, the scientists can put a number (or several numbers separated by a comma) in the corresponding Skills cell. Each vehicle has a corresponding list of numbers which identify the scientists in each vehicle (a scientist can go on multiple trips and so be in multiple vehicles). Vroom will use the skills feature to only send a vehicle to a destination if there is a scientist in the vehicle who can identify the specimen there.

The Time field is occasionally useful too. It’s an estimation of how long the vehicle will spend at the destination not moving (in seconds). In most cases, the scientists I was routing for would simply park the car, clip a branch of a tree or shrub, and get back in the car, so this was effectively zero. In some cases, however, the GPS point could be a mile or more from the road, meaning that a longer hike was in order.

The scripts that I’m making available are very much hacked together, not up to my usual standard for code I make public. For example, there are important constants in the file that have to be set or adjusted when the script is run. I didn’t come up with a convenient way to use argparse for this.

We have the already-discussed list of VEHICLES:

VEHICLES = [[1,2],[3,4]]

This identifies two vehicles / trips, where two scientists (numbered 1 and 2) are in the first, while two other scientists (numbered 3 and 4) are in the second.

We have lists of starting coordinates and ending coordinates for each vehicle. These are in [longitude, latitude] format because that’s what Vroom itself uses.

STARTCOORDS = [[-118.44, 34.068], [-118.44, 34.068]]
ENDCOORDS = [[-118.44, 34.068], [-118.44, 34.068]]

Note how the starting and ending coordinates for the vehicles are all the same in this case. That’s a common pattern (starting and ending at the university), but you’re not locked into doing that by any means.

Last we have TIMEAVAIL:

TIMEAVAIL = [400000, 400000] # seconds

This is the messiest part. It’s the total drive time allowed for each car. In theory, this would be perfectly known to the participants in advance (they could allow more time for stops, breaks, sleeping, etc., as needed). In practice, though, it’s a balancing act. The research trips have to be done, so even if finishing all of them would take more time than the scientists theoretically have, put together, that just means that they have to find more time from somewhere. So in practice the only workable approach that I found was setting this value unreasonably high for each car, then reducing it selectively to get routes that looked reasonable. When you can’t reduce it any more without the router failing, you have a roughly optimal solution.

Vroom will print a large JSON file with the results (and the routes themselves from OSRM in polyline format), and so I have another script that takes the result and prints the result as a long list of coordinates for each vehicle, and also exports the routes (as lists of coordinates) to text files, so that they can be shown on a map. Theoretically I ought to create KMLs or something more useful, but this turned out to be more complicated than I had hoped to do in Python and so I ran out of time.

My repository contains a file that shows how I run the program for my specific setup. Obviously you’ll need all the pieces in place (Vroom, OSRM on a server, etc) in place for this to work. Running the script, even for a large list of destinations, takes only 3-5 seconds, so re-running it while changing the parameters is no big deal.

The results are extremely nice looking:

A map showing a solution to the vehicle routing problem in 

Thanks to GPS Visualizer for making it easy to plot these routes.

This solution shows three different routes, constrained to a limit of about a week per trip. Several of the points have additional constraints, such that only one scientist can collect the sample, and this has still allowed for a nice solution. In many cases, it’s possible for human intuition to come up with a map that looks a lot like this one, but getting the exact order of coordinates right turns out to be crucial in saving hours of drive time.

At the end of the day, I suspect my exact approach won’t be completely workable for most people. You’ll have slightly different constraints than me, a different input format for your data, or some other problem. That’s why I’m just sticking these scripts online instead of spending the time to make sure they’re as functional and well-written as they could be. Hopefully, others will be able to look to them for help in solving their own routing problems.

Fixing a broken QR code in FEZ

13 January 2021

I’ve been playing FEZ for the first time recently, and came across this room where some NPCs are constructing a QR code on the wall:


This seemed very likely to be a puzzle. It’s almost a complete QR (most of what’s missing is the top left positioning square), and anything under the scaffolding can be reconstructed, so it stands some chance of scanning after a few fixups.

I imported the screenshot in GIMP. I combined it with another screenshot of the same scene to remove the characters, leaving only the scaffolding. I used the Color Exchange tool (Colors > Map > Color Exchange) to replace the lighter purple color in the background with white. I then used the Threshold tool (Colors > Threshold) to make anything else in the image black.

Then, in another layer, I created a grid precisely aligned with the blocks in the image to help me see and fill in the missing portions. GIMP has a tool for that (Filters > Render > Pattern > Grid). I replaced the partial blocks hidden by scaffolding with completed blocks. This result looked like this (the grid is at 25% opacity):


Unfortunately the result still wouldn’t scan, even when I added the third position symbol back to the top left. Too much seemed to be missing, at least for the apps I tried. But since this seemed to be a puzzle that the developers intended you to solve, I assumed there was something more here, so I broke out the QR code spec.

The first thing to understand about QR codes is that they’re relatively simple ways of encoding binary data (ones and zeros) using black (1) and white (0) blocks. Certain parts of the image are only used to align scanners: they don’t contain any data. Other parts contain format information: this tells a scanner what kind of QR code it is and how to decode it. I was hoping this QR code would contain valid format information, as otherwise I’d have to test all the possibilities. Here’s an illustration of the different parts of a QR code. Image credit: Wikipedia.

An illustration of a QR code

From the size (25x25), we know this is a V2 QR code, which doesn’t have version information. The first question to answer was whether the QR code still contained valid Format Information. This information is error corrected and duplicated within the image. It appears in the QR code in the sections highlighted in blue:


One of the Format Information bit strings seems complete: it combines the top-right portion with the bottom-left portion. I’m working with the assumption (that I might need to change later) that anything not in the top left quadrant is “finished” by the NPCs, meaning it’s trustworthy. Hopefully this means I won’t need to apply error correction to the Format Information.

Information in QR codes is masked (XORed with a known pattern) to break up large chunks of white or black that give scanners difficulty. The Format Information uses the static mask of “101010000010010”1. Wikipedia incorrectly says this mask is “101011001010101”.2 The data in the QR code will use a different mask, chosen dynamically by the encoder so as to minimize large white or black chunks.

Here is where I began to be surprised by how far I could go using only GIMP. I expected to be importing the data into Python relatively quickly, but being able to toggle masks on the QR code directly and observe the result visually turned out to be very handy.

Using the new “Exclusion” layer mode, you can selectively invert parts of the underlying image using a black and white layer. Take care to observe the correct bit order when creating the mask (MSB: for the Format Information, that means bottom to top, left to right), as seen here:


Applying the mask, we see the following:


Note that because of the way the GIMP Exclusion layer works, we have to use white as the “on” bit for the mask. Black would be more appropriate since that’s what the QR standard uses for an “on” bit.

According to the spec, the first two bits give the error correction level, and the third through fifth bits give the data mask information. That’s once again in MSB order, so bits 14 and 13, and 12, 11, 10, respectively. In this case, we have “11” for the error correction level, and “001” for the data mask. That tells us that we have error correction level “Q” (which we’re ignoring for now), and “i mod 2 = 0” for the mask (this is one of several options for a mask that is chosen by an encoder). This means that the mask is active (flipping the underlying data bits) for every even number row, starting at 0 for the top row.

We can quickly create such a mask in GIMP by making a small portion of it (just two blocks, in this case), then expanding that with Filters > Map > Tile. We then trim out any portion that is part of a position symbol, timing symbol, alignment symbol, quiet zone, or Format Information (so that only the data will be masked). The spec (and other references) are pretty clear about what these areas are, so I’ll skip to the result here:


We’re now ready to begin trying to read the raw data! The order of bits is detailed in section 8.7 of the spec. We start with the bottom right block, then go left, then go back and up, then left again, reading upwards in a step-wise pattern. When we reach the position symbol at the top we move to the 2 block wide column to the left, and this time step down. (Discovering that QR code data starts with the bottom right and that it ends with error correction data gave me hope that the QR code was decipherable.) The reading order for the intricate sections is given precisely in the spec. I’ve indicated the interesting parts below.


Section 8.4 of the spec indicates that the data stream is prefixed by a header, where the first four bits are a “mode indicator”, followed by a length indicator. If you’re just encoding raw binary data in a QR code, you may as well just use 8 bit bytes, which is mode “0100”. But if you’re representing numeric or alphanumeric data, you can encode it more efficiently. The FEZ QR code begins with “0010”, which indicates an alphanumeric QR code.

This means that the next 9 bits (for a V2 QR code) are used to give the number of characters in the data. The next 9 bits of our QR code are “000010111”, which is the number 23 represented in binary.

For maximum efficiency, alphanumeric mode encodes 45 possible characters into blocks of 2 characters per 11 bits. (45^2 / 2^11 ≈ 98.9%, which is pretty good efficiency). To access the characters, we divide the 11 bit integer by 45. The result without the remainder is the first character, and the remainder is the second character.3

Each number represents a character according to its position in the following string:


Note that we have an odd number of characters, so to save every bit possible the last character will be encoded as a single 6 bit value, instead of wasting another 5 bits.

With all this information in hand, we can transcribe the data from the QR code. Anything after the last piece of data will be part of the error correction bits, which we’re hoping we don’t need. Now that we know how much data there is (22 × 11 / 2 + 6 bits), we can also identify each portion of the QR code by its function:


Blue represents Format Information (as before). Green represents data. Red represents error correction. Anything purely black or white is a static part of the QR code.

I have transcribed the data like so:

10011011100: 27 29 (R T)
11001101111: 36 27 (Space R)
10100111101: 29 36 (T Space)
01111001110: 21 29 (L T)
11001101111: 36 27 (Space R)
10100111101: 29 36 (T Space)
01111001110: 21 29 (L T)
11001101001: 36 21 (Space L)
10100111101: 29 36 (T Space)
01111001110: 21 29 (L T)
11001101111: 36 27 (Space R)
011101: 29 (T)

In other words, the data says “RT RT LT RT LT LT LT RT”. This is a sequence of inputs you can make with the controller in FEZ. You usually receive a collectable for doing so.

I also wrote some Python code to do the decoding automatically, which you can find here:

With all the information we now have, we can actually generate the final, corrected version of the QR code. To do that, we’ll need a library that allows us to set the parameters manually (because an automatic QR code creation tool might choose a different mask or error correction level, giving a completely different result). I used the one here. Exporting the resulting SVG and layering the result on the original, we can see that it’s nearly a perfect match!


Having finished all this, I reopened FEZ, and entered the input sequence. And NOTHING HAPPENED. I tried multiple times to make sure I wasn’t screwing anything up.

I looked up the puzzle online, and apparently a completed version of the QR code is in a later stage of the game, and this unfinished one is just for “lore”. I find this explanation frustrating, because it clearly seems to have been designed so that an enterprising person could reconstruct the code themselves. Everything you need is there. Yet I have to admit… it doesn’t work. You can’t enter the code in this room, you have to enter it elsewhere.

I think this is a mistake on the game designers’ part. If a puzzle gives you everything you need to solve it, the solution should work. It’s rather frustrating to put that much effort into solving a puzzle only to find out that your effort wasn’t anticipated, and the puzzle is actually solved for you later in the game.

  1. According to section 8.9 of the QR code specification. ISO/IEC 18004:2000. 

  2. As of 11 January 2021. 

  3. It’s pretty easy to understand how this works. In a base 10 system, you can “encode” 2 characters out of a set of 10 possible characters in a 2 digit number: “12” for example. Dividing by 10 and ignoring the remainder gets you back the first character (“1”), and dividing by 10 and taking the remainder gets you the second character (“2”). This system works exactly the same, it’s just that it’s a base 45 system. If we used base 45, you’d be able to just read off the two characters after printing the number. 

How the pursuit of HD video ruined TV for half a decade

16 September 2020

I’ve been watching a lot of early-2000s sci-fi TV recently, and I’ve noticed that just about every show is plagued by a very specific problem.

Take Stargate SG-1 as a pretty typical example. According to an interview with one of its producers1, the first two seasons of the show were shot on 16mm film, and then until season eight the show was shot on 35mm film. It was finished along with the effects compositing in standard definition, of course. For most if its runtime it was a full screen show as well. All this, of course, means that the first seven seasons of the show aren’t that great looking by today’s standards.

An image from Stargate SG-1 Season 7 Episode 21, "Lost City"

By 2004, however, audiences and networks wanted to see shows in HD, and this meant a different set of expectations and trade-offs. Stargate SG-1 moved to HD digital acquisition for its eight season. The results are, well…

An image from Stargate SG-1 Season 8 Episode 17, "Reckoning", showing blowout

The choice of this image is obviously selective. What’s happening here is that the dynamic range of the scene (the difference in luminance between the darkest part of the image and the brightest part) is too great to be captured by the camera. The highlights are blown out. This scene is one of the worst looking in the show, but as several other shots indicate, the same problems are obvious in many other scenes too.

An image from Stargate SG-1 Season 10 Episode 8, "Memento Mori"

An image from Stargate SG-1 Season 9 Episode 18, "Arthur's Mantle"

In addition to frequent blow-out issues, these HD shots suffer from quite a bit of dynamic range compression and loss of detail in the highlights. This is visible as a kind of “pastelization” of bright colors, making them “flatter” looking than they ought to be, even though their saturation is at a normal or even raised level.

While the increase in resolution is obviously appreciated, it also comes with quite a bit of quality loss in terms of accurate, film-like colors. Hair and faces tend to glow white or even bluish under any bright light, colors are harsh and crushed, and many shots now have an ugly “plastic” look to them that further exacerbates issues with the CGI and other effects shots. Even though many of these shots wouldn’t look photorealistic without effects, the absence of photorealistic images makes the effects shots feel worse on a subconscious level, the plastic glow giving the show a queasy fever-dream veneer.

Primarily, these issues effect shots in uncontrolled outdoor lighting. However, SG-1’s sister show, Stargate Atlantis, is frequently affected by lighting issues even when shooting indoors.

An image from Stargate Atlantis Season 3 Episode 2, "Misbegotten"

An image from Stargate Atlantis Season 3 Episode 12, "Echoes"

I’m not sure what the issue here was. Perhaps it can be chalked up to different crews and the producers having their primary focus on the last season of SG-1, airing around the same time. However, I’d also point to the Atlantis crew’s unwillingness to compromise on the shots. The SG-1 outtakes I’ve used as examples are generally exceptions — what the show looks like when it looks noticeably bad. The weird plastic-like appearance of HD remains on all the shots, but they usually managed to avoid too many issues with the highlights.

I don’t have firm figures on how much the filming changed due to HD, but it’s pretty indicative that I had to look through quite a few episodes of SG-1 before I could find the kind of shot I was thinking about. Many episodes from the last three seasons are shot entirely inside with carefully controlled lighting, on some combination of the Stargate Command set and a few others. As I remember it, this marked a clear shift from the kind of “new planet” episode openers many of the earlier seasons depended on.

Even when shots in the later seasons of SG-1 are clearly outside, they tend to be in places where the characters are in shadow, for example in the ubiquitous forest scenes, inside a warehouse, or on the shadowed side of a building or mountain. Anything to control the bright highlights! In other scenes, heavy lighting was used to blend the scene together better and give it a unified aesthetic, removing any harsh glows.

An image from Stargate SG-1 Season 8 Episode 16, "Reckoning Pt. 1"

Quick trivia question: what do Attack of the Clones, Stargate SG-1, and Battlestar Galactica all have in common? Answer: they were all shot on the same camera, the Sony F900 HD series. This was an early professional HD camera which was pretty readily available and affordable even for TV shows. It suffers from a number of issues, including obviously the dynamic range issue, mentioned above, but also the fact that anyone working with HD video this early was most likely shooting in the same gamma (e.g. Rec. 709) as the final product, instead of shooting with a log curve as more modern productions would. Even if the image sensor in the F900 had been fantastic, shooting this way would probably have crushed quite a bit of the original detail out of the image, in a way that couldn’t then be recovered in post-production.

To be clear, the F900 wasn’t unique in having these problems, they were largely a side-effect of being an early adopter of HD acquisition. That said, while many episodes of SG-1 took care to avoid putting the worst aspects of HD video on clear display, Attack of the Clones and Galactica didn’t work with these limitations, and the results are quite frequently terrible.

An image from Attack of the Clones

An image from Attack of the Clones

An image from Attack of the Clones

The dated CGI of Attack is often credited for giving the film its plastic look, and that’s certainly contributing given the number of shots that are almost entirely greenscreened, but as you can see from examining these shots closely, the lighting issues at play with this early HD camera is another significant factor.

But now of course we come to Battlestar Galactica, which more than any other show I can think of illustrates the pitfalls of shooting with HD video cameras.

An image from Battlestar Galactica Season 1 Episode 2

An image from Battlestar Galactica Season 1 Episode 2

Frankly, Battlestar Galactica is so bad it seems deliberate. Almost every single shot in the show looks like this, it’s borderline unwatchable.

An image from Battlestar Galactica Season 1 Episode 13

An image from Battlestar Galactica Season 1 Episode 12

Unfortunately, it does in fact seem to be the case that some of this was intended, as part of the creators playing with what to them was a new format. An article on the use of HD2 in Star Trek Enterprise and Battlestar Galactica describes the difficulties that the creators of Galactica faced when they tried to achieve the right aethestic for the show, which was “grungy”, “gritty”, with pushed grain. To try to get this result on a TV shooting schedule, DP Steve McNutt developed a realtime color management process for adjusting color on set while using little, if any, post-production color correction.

As part of an attempt to replicate a “documentary-style”, shot-on-film look, McNutt “[pushed] for video noise to emulate grain”, sometimes as high as +18dB. The visual effects supervisor, Gary Hutzel, says that the DP “pushed gamma, crushing the blacks, clipping the highlights”. That’s extremely telling. It’s not worthwhile to try to parcel out blame, in hindsight, for a show shot almost 20 years ago; what’s important however is how the specific limitations of shooting in HD forced creators to experiment with novel techniques to try to get the specific looks they wanted to achieve, with expectations often keyed to the behavior and appearance of the film stock they were used to working with.

Even with the F900, Battlestar Galactica didn’t have to look as bad as it did, but it’s easy to see how seeking the intensity and immediacy of a grainy film stock led too many directors astray.

You might be thinking of the Battlestar Galactica mini-series that aired the year before right now, and wondering why it looked so good. Indeed, it does look fantastic. It was shot on 35mm film.

An image from Battlestar Galactica Miniseries Pt. 1

An image from Battlestar Galactica Miniseries Pt. 1

An image from Battlestar Galactica Miniseries Pt. 1

An image from Battlestar Galactica Miniseries Pt. 1

Now you might recall that the Battlestar Galactica miniseries is available on Blu-ray. And it’s not upscaled — it looks great. So if everyone was already shooting on 35mm film before HD video took hold, why not just continue doing so when the networks wanted HD?

It turns out that most producers would have liked to stay on film, but were forced to change due to budgetary demands. For example, SG-1 producer John Lenic said (speaking about follow-up show Stargate Universe):1

Film still is from a look perspective and in my opinion, the best looking format to shoot on. … The only reason that we didn’t go film is financial. Film still is roughly $18,000 more per episode than digital as you have all the developing and transferring of the film to do after it is shot.

This transfer cost was something that producers were willing to bear when it seemed to be necessary, but less so when HD promised a sharper image for less money. In particular, continuing to use film when shooting for HD would have meant having to do the transfers in HD as well, along with the same finishing costs required for HD post-production, including the effects shots. Additionally, the “cleaner” image of HD video, lacking in rough grain from film stock, made effects compositing a bit easier to do in HD.

With Battlestar Galactica as well, director Michael Rymer initially opposed2 shooting with HD cameras. Here too, the reasons were financial, as it wouldn’t have been possible to produce the show at all if it were shot on film. In the previously cited article, Rymer indicates that he noticed many of the issues that I have pointed out. In particular, he says “the video was picking up the fluorescent bars of various consoles on our [spaceship interior] set”, and calls daylight exteriors “less than satisfactory” and “the worst environment for HD”. That said, Rymer did eventually come around to the look of the show on HD, saying that the sensor noise “approximated film grain nicely” and that he could ultimately achieve the desired aesthetic for the show. On this point I have to disagree, but again, we’re looking back with 20 years of hindsight on how badly early HD footage and CGI work has aged.

As it turns out, a ton of productions were shot with the F900 camera, and cameras like it, and the effect on TV production quality was fairly devastating. This isn’t a critique of shooting digitally in general, of course; the technology has vastly improved over the course of the last decade. Some shows, like The Big Bang Theory, are so reliant on internal sets and carefully controlled lighting that the limitations of shooting HD don’t really show. Others, like The Office, make use of the raw harsh lighting to their advantage, creating an alienating and sterile environment that pushes the presence of the camera in your face, achieving a documentary tone that in my opinion Galactica never did.

Once you decide to go with digital acquisition and approximate the final color grade in-camera, there’s basically no hope of an improved release (for home video or otherwise) years down the road. On the other hand, it’s worth pointing out that early seasons of SG-1, because they were shot on film, could be released in HD one day. The difficulty of course is to find someone willing to put up the money to have the original negatives scanned, and possibly recomposite the effects in HD. For very popular shows, like Star Trek: TNG, this has actually been done, but there’s little hope of the same for less popular shows like Stargate.

Here too, then, there are tradeoffs. While we do have HD versions of the later seasons of SG-1 (though only on streaming services, not Blu-ray), we have to deal with the permanent damage to the image quality done by the acquisition method. On the other hand, the earlier seasons have the potential to see beautiful high definition scans, but this will likely never happen because it’s simply too expensive.

Bart Ehrman and the “strictly historical point of view”

12 September 2020

I recently read the book Jesus, Interrupted by Bart Ehrman. It’s a very nice book. However, he has the following very strange thing to say about the method historians follow: “If historians can only establish what probably happened, and miracles by their definition are the least probable occurrences, then more or less by definition, historians cannot establish that miracles have ever probably happened.”

This is the sort of statement that causes me to scratch my head and re-read several times because I get the strong feeling that it has to be wrong. Ehrman claims that historians must accept methodological naturalism. In other words, a historian can never conclude on the basis of their evidence that a supernatural event occurred. They must always conclude (whatever their personal convictions) whatever is most likely, and what is most likely by definition cannot be a miracle.

While this requires some unpacking already, it seems to me to be based on seriously mistaken notions of both supernatural events and likelihood itself. Before getting into that I think it’s worth considering the implications of such a view.

If I read Ehrman correctly, should he miss the second coming, even extensive video evidence of Jesus Christ returning through the clouds to rule the nations would not convince him of its happening. A supernatural event of any kind is simply above the pay grade of historians to comment on. These events are purely the realm of faith. (One wonders if he would believe his own eyes.)

Not only is this odd, it’s already directly in conflict with Ehrman’s own claims about other supernatural events. For example, this excerpt I’m commenting on is from a book which argues “if the findings of historical criticism are right, then some kinds of theological claims are certainly to be judged as inadequate and wrong-headed. It would be impossible, I should think, to argue that the Bible is a unified whole, inerrant in all its parts, inspired by God in every way.”

Let’s consider what the evangelical claim that the Bible was verbally inspired by God entails, counting the instances of miraculous activity. Conservatives claim that each word of the Bible was inspired by God in the original autographs (1), the books were sufficiently preserved by scribal copying over time (2), and the correct books were canonized by the winning side in the theological battles of the early church (3). Ehrman is on the record as thinking these claims are untenable in the light of the historical evidence (e.g. in light of contradictions in the text), and in fact has dedicated this very book to proving that point. How is this possible if all claims of miracles are the proper domain of faith and not history?

In the book, Ehrman also pokes some well-deserved fun at the fundamentalist KJV-only movement. But what this sect claims, on any intelligible version of their views, is that God inspired the work of the translators. If this is a faith-claim, as it certainly must be, what are we to make of Ehrman and other historians hastening to point out that the translators of the KJV based their work on inadequate sources and made many mistakes? Why should that matter?

To try to clarify this, let me quote extensively what Ehrman has to say about miracles.

Miracles, by our very definition of the term, are virtually impossible events. Some people would say they are literally impossible, as violations of natural law: a person can’t walk on water any more than an iron bar can float on it. Other people would be a bit more accurate and say that there aren’t actually any laws in nature, written down somewhere, that can never be broken; but nature does work in highly predictable ways. That is what makes science possible. We would call a miracle an event that violates the way nature always, or almost always, works so as to make the event virtually, if not actually, impossible. The chances of a miracle occurring are infinitesimal. If that were not the case it would not be a miracle, just something weird that happened.

I think this quote points up the problem quite nicely. Ehrman is begging the question by slipping in the assumption of naturalism into what ought to be a defense of methodological naturalism. Ehrman’s definition of a miracle is quite acceptable, actually. I’ll quote it again: “[a miracle is] an event that violates the way nature always, or almost always, works…” His conclusion from this is that “The chances of a miracle occurring are infinitesimal.”

This simply does not follow. One cannot simply assume that “the way nature … works” is the same as what actually happens. Miracles are by definition supernatural occurrences! If you assume that the story of what happened at some point in the past can be filled out entirely by causal or quasi-causal chains in the natural order of the universe, you’ve simply assumed that miracles do not happen. That, of course, assumes far too much for Ehrman.

What Ehrman can safely conclude is that the chances of a miracle occurring naturally are infinitesimal. To get from there to the claim that historians are justified in rejecting these explanations in all cases requires much more work. I can think of several ways to do so.

For example, maybe Ehrman wants to position the historian as a sort of scientist, who can’t consider supernatural occurrences precisely because they’re supernatural — they’re in the wrong domain. On this view the job of the historian is to reconstruct the best possible naturalistic explanation for what happened in the past.

One problem with this interpretation is simply that Ehrman doesn’t seem to be saying this. He repeatedly asserts that the problem with miracles for the historian is their unlikelihood. He writes:

Historians can only establish what probably happened in the past. They cannot show that a miracle, the least likely occurrence, is the most likely occurrence.

The bigger problem with this view is that it’s just a silly and arbitrary restriction of the historian’s task. Surely the job of the historian is not like that of the scientist at all! The scientist must construct the most plausible account of the behavior of the natural world. The historian, on the other hand, is tasked with giving the most plausible account of what actually happened in the past.1 If the most plausible account involves some supernatural activity (recall the example I gave of video evidence), so be it.

I think the most likely explanation for Ehrman’s reticence to consider supernatural explanations is that they’re an enormous unknown. In order to get from the claim that miracles are extremely unlikely to happen naturally to the claim that they’re extremely unlikely, one has to have a prior for how likely non-natural events are. In other words, do miracles happen? It is not impossible to imagine a world in which miracles happen all the time; indeed, some evangelicals believe we live in such a world. Accounts of miraculous healings and near-death trips to heaven appear regularly in media targeted at conservative Christians. Ehrman (as an agnostic) does not take these accounts seriously. He (and I) don’t believe we live in a world where miracles are common. That said, there’s an enormous difference between the claim that miracles are (at least) uncommon, and the claim that a miracle is always “the least likely occurrence”.

This leaves a vast gray area. For Ehrman, we have no convincing evidence of miracles that would lead him to set a high enough prior for them to figure in many historical explanations. On the other hand, he is unwilling to take the materialistic stance that miracles are impossible and always ruled out as possible accounts of what really happened. Thus he tries to bracket off these concerns as completely as possible from the historian’s task.

If this is really the best reading of what Ehrman wants to do, it’s hard to have methodological objections. It would be an enormous issue for historians if at every turn they had to speculate endlessly about the appropriate prior likelihood of supernatural intervention in the normal course of the universe. Given that miracles are at least not every-day occurrences, historians seem to be justified in attempting to find straightforward scientific accounts of past events. I have no objection to this.

What I continue to object to is the specific defense offered by Ehrman. He writes about the resurrection of Jesus, as a purported historical miracle:

The resurrection is not least likely because of any anti-Christian bias. It is the least likely because people do not come back to life, never to die again, after they are well and truly dead.

In other words, he follows up his claim that his assignment of low probability to the resurrection is not the result of anti-Christian bias with what’s probably the only blatant expression of anti-Christian bias in the whole book! If it is simply a fact that people do not come back to life, then Jesus did not come back to life.2 No wonder the resurrection has such a low probability in his estimation!

What Ehrman presumably means is that under some unclear historian-centric notion of probability, the probability of the resurrection of Jesus is extraordinarily low. This is a notion of probability that makes naturalistic assumptions, not because the probability of supernatural events is low (which would beg the question), but because it is appropriate for historians to bracket off these types of considerations. In other words, it is because the likelihood of a supernatural explanation’s being true is indeterminable that historians must steadfastly refuse to speculate about them.

This passage remains remarkably unclear. If something like this is what Ehrman means, he would do well to say so directly. Meanwhile, those interested in more general answers to these questions must consider the matter more holistically. What one has to think about is quite simply the plausibility of two more or less complete descriptions of the entire world. Which makes more sense: the view that miracles sometimes occur and are the best explanations for some past events? Or that there have never been miracles and all past events are explained in a naturalistic fashion? This is a difficult question to answer — but it is not a question from which reason must be banished as a matter of faith.


After writing this, I subsequently read one of Ehrman’s other books, How Jesus Became God. In this book, written about five years after Jesus, Interrupted, Ehrman has a more nuanced take on the historian’s role. For example, Ehrman writes,

It is not appropriate for a historian to presuppose a perspective or worldview that is not generally held. “Historians” who try to explain the founding of the United States or the outcome of the First World War by invoking the visitation of Martians as a major factor of causality will not get a wide hearing from other historians—and will not, in fact, be considered to be engaging in serious historiography. Such a view presupposes notions that are not generally held—that there are advanced life-forms outside our experience, that some of them live on another planet within our solar system, that these other beings have sometimes visited the earth, and that their visitation is what determined the outcome of significant historical events.

This is a useful comment. Something about the historian’s task prevents them from invoking beings or phenomena that are not accepted already by a majority of other historians and scientists. In perhaps the best version of his view in the book, Ehrman continues:

The supernatural explanation, on the other hand, cannot be appealed to as a historical response because (1) historians have no access to the supernatural realm, and (2) it requires a set of theological beliefs that are not generally held by all historians doing this kind of investigation.

Here Ehrman drives the cleanest wedge between the question “what actually happened?”, and the historian’s question, which is (here) seemingly “what is the most plausible historical-naturalistic reconstruction of what happened?” This confirms that my previous suggestion that Ehrman wants to bracket off supernatural concerns from history may have some truth to it.

Unfortunately, Ehrman is not always so clear is this book. Later, he returns to making almost exactly the same claim that he made in Jesus, Interrupted:

But simply looking at the matter from a historical point of view, any of these views is more plausible than the claim that God raised Jesus physically from the dead. A resurrection would be a miracle and as such would defy all “probability.” Otherwise, it wouldn’t be a miracle. To say that an event that defies probability is more probable than something that is simply improbable is to fly in the face of anything that involves probability. Of course, it’s not likely that someone innocently moved the body, but there’s nothing inherently improbable about it.

Here Ehrman returns to the concern with probability and the strange claim that miracles are inherently the most improbable explanation. This suggests, contra the statements made earlier in the same chapter, that the historian is concerned with the question “what actually happened?” and is simply inferring to the most plausible explanation. As I argued above, anyone truly committed to answering this question cannot simply forswear any non-naturalistic explanations, because there’s simply no way to show a priori that the supernatural will never figure in the most reasonable account of historical events.

  1. As philosophers of science have pointed out, there is some overlap between the fields in Biology. 

  2. Paul noticed this with remarkable clarity. “Now if Christ is proclaimed as raised from the dead, how can some of you say there is no resurrection of the dead? If there is no resurrection of the dead, then Christ has not been raised; and if Christ has not been raised, then our proclamation has been in vain and your faith has been in vain.” (1 Cor 15:12-14 NRSV) The argument here is straightforward and if Paul is right, then the assumption that people do not come back to life is certainly an anti-Christian one. Likewise, Paul would no doubt be surprised to hear that “Believers believe that all these things are true. But they do not believe them because of historical evidence.” Paul (and the Gospel authors) regularly offer such evidence, as Ehrman himself points out in the book. * * 

Next page
©2021 Adam Fontenot. Licensed under CC BY-SA. About Me Projects RSS Feed