Hi! I’m a graduate student in philosophy. Thank you for visiting my website, which hosts my personal blog. If you would like professional information or to contact me directly, please visit my about page.
I’ve used LibreOffice for quick word
processing tasks for years now — since before it was forked from
OpenOffice in fact. However, for years I’ve drafted
documents in another tool, before copying them into LibreOffice for
the final export to PDF. The reason is that the font rendering in
LibreOffice is infamous for how terrible it is, with a large number
of issues reporting going back years.
Here’s a brief explanation of the problem. Fonts set the amount of
space (called “tracking”) around each
glyph. Certain combinations
of characters do not look right with the default amount of space,
and a feature called pair kerning allows a text renderer (like
LibreOffice) to adjust this space for individual letter pairs. For
instance, the large open bowl of the uppercase “C” looks too empty
by default, so the next letter is often brought closer to it.
Unfortunately, LibreOffice’s rendering on low resolution screens was
so bad that enabling pair kerning (it has been the default for many
years) made it worse. In some cases the failure is catastrophic:
This is a real screenshot taken on my computer. I wrote software to
upscale the image so that the problem is clear. Each individual
(red, green, or blue) sub-pixel as shown on my screen is being
re-rendered at nine times its original size, in grayscale. This
helps to capture the effect of
which uses the shape of the pixels on modern LCD screens to triple
the effective horizontal resolution of the text. Here is the
original screenshot, for comparison:
Pair kerning is implicated here because it is actually this feature,
when enabled, that turns “California” into “Califomia”. The font
shown here is Liberation Serif 12pt, at 120% page zoom. Hinting is
Why is text rendering on LibreOffice 7.3 so bad?
The problem was caused by an approach to text rendering that
predates the introduction of the codebase into version control in
the year 2000. Back then, text renderers were rather stupid, and did
not do a good job of rendering strings of text in a readable way on
the screen. LibreOffice (then OpenOffice) therefore did its own
LibreOffice knew where the glyphs should go (called the printing
position, because it was where the glyphs would be placed when sent
to a high resolution output like a printer), but just sending these
“ideal” locations to the text renderer would result in subpar text.
On the other hand, if LibreOffice rendered the text at the preferred
screen position, the width of the text would not match up with the
printer’s output (because the screen metrics of the font differ, e.g.
due to hinting).
LibreOffice is a WYSIWYG (what you see is what you get) editor, so
this is unacceptable. Lines that exceed the word processor’s margins
must reflow, so you would end up with cases where the printed
document would have different lines than the document as shown on
Instead the authors adopted a
of each glyph’s placement is determined by the ideal position
according to the screen metric, and 25% is determined by where the
glyph would be placed if its right border was aligned with the right
border of the ideal printing position. WYSIWYG is made possible by
realigning the text with the printing position every time white space
appears in the text.
Unfortunately, the end result is not very good by the standards of
a decade later. On a low resolution screen like mine, pixels are
quite large relative to the size of the glyphs:
The above shows close to ideal kerning for the r/n letter
combination for this font at 12pt on my screen. As you can see,
being off by one pixel is enough to make “rn” into “m”.
Despite LibreOffice making its best effort to place glyphs
intelligently, most of that work was thrown away at the last stage
in the rendering process, because the glyph positioning engine did
This is a different concept than sub-pixel anti-aliasing. While the
font renderers used by LibreOffice have generally supported the
latter, sub-pixel positioning support is much more recent.
Sub-pixel anti-aliasing (sometimes more generally called sub-pixel
rendering) is the use of the thin red, green, or blue components of
LCD screen pixels to improve the resolution of text. Sub-pixel
positioning uses anti-aliasing (potentially including sub-pixel
anti-aliasing) in order to render text at positions more precise
than a single pixel.
This is crucial to the proper appearance of text. Because pixels on
low-resolution screens are so (relatively) large, rounding the
location of glyphs to the nearest pixel generates consistently bad
results. Combine that with pair kerning (which tends to move glyphs
closer together, and also changes the heuristic described above to
favor the printing position), and you’re asking for catastrophic
What does LibreOffice 7.4 do differently?
For at least the last 22 years, LibreOffice has done its own glyph
positioning under the assumption that the underlying font
rasterization engines were not capable of creating high quality text
on LCD screens when passed the “true” positions of each character
(i.e. the printing positions).
This was true for most of LibreOffice’s history. After all, while
rendering has been acceptable on Linux for at least a decade (with
the proper settings and a recent enough FreeType), commonly used
libraries like Pango / Cairo did not support sub-pixel positioning
Qt was ahead of the game for a long time,
supporting it since some time around 2015.
During the LibreOffice 7.4 development cycle, Caolán McNamara
with this behavior, initially adding an option to just kick the
printing positions down to the rasterizer and let it place them on
the screen using its own algorithms. The result was successful
enough and fixed so many
that it was made the default mode in LibreOffice 7.4.
Because the underlying rasterizers now support sub-pixel
positioning, this change results in LibreOffice getting that for
free, in addition to the intended change of eliminating the buggy
positioning code from the render path.
The results are breathtaking:
It’s rare to see such an enormous improvement in a piece of software
in a minor version bump, and this one appears to have gone almost
unnoticed. It’s even rarer to see an improvement like this with a
simplifying change, in this case one that makes a bunch of old code
My thanks to Caolán McNamara for working on this, and for
backporting the patches to LibreOffice 7.3 at my request.
LibreOffice is now a viable choice for me when drafting documents.
I hesitate to share the following before and after picture of some
text at the original size for fear it may be misconstrued. Don’t
expect this text to look great on your system, even if you view it
at 100% zoom (which you should — right click to open the image in
a new tab). Font rendering happens at a very low level, relying on
details like screen DPI and pixel orientation that may vary from
screen to screen. This is also a bit of a stress test, rendering
some serif fonts with many details on a low resolution device at
Rather than deciding which rendering looks better to you overall,
focus on the differences in glyph placement. Where are the glyphs
too close together? Where are there ugly gaps? To my eye,
LibreOffice 7.4 has significantly reduced severe positioning errors
(“met” in the fourth line of the Garamond sample) while achieving
superior rendering even in cases where LibreOffice was already
acceptable (every “th” pair is slightly too close in Caslon), thanks
to sub-pixel positioning.
Content note: this article contains a discussion of weight loss and healthy
eating, and includes my before and after pictures at the end. You can view a
version of this document without these pictures here.
I lost 80 pounds following a diet focused on calorie restriction and satiety by
writing software to track my weight and suggest adjustments. The program is
and it is available as a free download for Windows or Linux.1
The premise of the diet is that it’s possible to achieve a precise
rate of weight loss when restricting your intake without counting the
calories in all the food you eat or knowing your daily energy expenditure. A
reasonable estimate of how much you’re eating is good enough, when assisted by
Because a calorie deficit and the resulting rate of weight loss are
mathematically related, software that measures your average rate of weight loss
can accurately determine whether you’re eating too many or too few calories,
even if you don’t count them exactly. You can use the information the software
provides to hone in on your desired rate of weight loss by making small
adjustments to your diet over time.
If this sounds interesting to you, read on.
People who want to follow this diet should talk to a doctor beforehand.
Why everyone is gaining weight and why it’s so hard to lose it
Some people believe that their bodies are genetically programmed to reach and
maintain weight at a particular “set point”.
This clearly can’t be the whole story. In the last 40 years, the average weight
of the adult population in most western countries has increased significantly.
If these weights reflect our natural set points, then we are forced to conclude
that just about everyone else who ever lived somehow never reached their
body’s natural weight. We’re the lucky few — except that apparently we’re
unlucky, since this supposed biological imperative just so happens to be bad
Yet neither can the set point view be completely wrong. It accords too well
with several common experiences.
When someone in a country without a fast-food culture moves to one with an
urban environment, they frequently gain a moderate amount of weight and then
stabilize at a new point from which they find it difficult to gain or lose.
Similarly, people who do manage to lose weight often hit plateaus — low
points at which they find further weight loss to be much more difficult. Some
people regain lost weight until they’re about where they started. What’s
I suggest that the truth lies somewhere in between. You do have a distinct
set of variables that dictate the weight you will approach in the long term.
These include your genetic makeup (with its effects on your metabolism,
skeletal structure, and so on), age, and hormone levels. But these variables
also include diet and exercise, two variables that are partially under your
In the rest of this article, I will refer to the weight you are approaching
given your current daily intake (and other variables) as your “equilibrium
weight”, to distinguish this from the simplistic “set point” view.
Holding everything else constant, if you are currently maintaining your body
weight and you permanently reduce your intake by 100 calories per day, you
will eventually stabilize at a new equilibrium point about 10 pounds lighter
than your current weight.2 A cut of this size is equivalent to drinking one
fewer can of Coca-Cola each day at work (five days a week).
Why losing weight is hard, and maintaining weight loss is even harder.
There’s one additional but very important wrinkle: one of the variables that
dictates your equilibrium weight is your personal history of weight changes,
because of its effects on your metabolism and body composition.
Consider three genetically identical people of my height and age. They
all weigh 235 pounds.
Person 1: has weighed 235 pounds for many years
Person 2: recently lost weight from 350 pounds
Person 3: recently gained weight from 150 pounds
According to one model of diet performance3, they will need to eat as
follows to maintain their present weight:
Person 1: 3050 calories
Person 2: 3000 calories
Person 3: 3400 calories
In other words, if all three want to lose weight, the person with the thin
“history” has an advantage. Because their body has not yet adjusted to
carrying this extra weight, they will burn an extra 400 calories a day for
A 400 calorie per day difference is a whopping huge one. It’s about the
equivalent of eating a pint of your favorite Ben and Jerry’s ice cream every
Monday, Wednesday, and Friday. On top of that, the previously 350 pound
individual was able to eat about 3800 calories a day at maintenance. So just to
maintain their current level of weight loss, they have to get used to eating
800 fewer calories than they are used to.
This may seem unfair. These are people who are genetically identical, the same
age, the same height, and the same weight. But one of them will maintain at
400 calories a day higher than the other. It is a little unfair, but remember
that these are ultimately temporary metabolic effects. Eventually Person 3,
who gained 85 pounds, will become Person 1, who has carried that weight for a
long time.4 The body doesn’t provide you with a metabolic advantage forever.
This leads to two important principles for maintaining weight loss:
Until your body becomes similar to that of a typical person who maintains at
your new weight, you will face transitory metabolic effects that resist your
weight loss. This means that until your body adapts, you will have to eat
less than others of the same weight can eat.
When you lose weight, you become a smaller person. Even once your metabolism
stabilizes, you will need fewer calories to maintain your weight than you did
at your original body weight. Your hunger systems may have accepted dieting,
but when the time comes to maintain they may be slow to adapt to your new
maintenance intake, and you may find them in revolt.
Most people ignore one or both of these principles. It’s tempting to treat
diets as one-time interventions similar to a medical procedure. This simply
will not work, as the foregoing discussion indicates. You can’t eat like an
overweight person who has temporarily starved themselves to fit into a smaller
pair of jeans. What you must do if you want to keep weight off is to
permanently eat like the thin person you now are. You must lower your
equilibrium weight through consistent, long term improvements in diet and
Most people recognize on some level that when they regain weight after dieting,
it’s because they “ate too much again”. What many do not realize, however, is
that “too much” means something different after dieting than it did before.
The worst part about all this is that you’ll begin to feel the effects of the
two principles while you’re still trying to lose weight.
Most people start a diet expecting to lose weight at a relatively quick rate
— and they do their first week or two. Unfortunately, most of this weight
loss is fake. Running a calorie deficit means that your body burns through its
store of glycogen, which is what causes it to tap your energy reserves (fat).
Along with the loss of this weight, you will see a one-time drop that’s just
water weight; your body has less food temporarily stored in your stomach and
intestines as it’s being digested. Your body will also adapt to reduced intake
relatively quickly, and your metabolic rate will drop slightly.
Metabolism is not magic. Your body obeys the law of conservation of
energy when your metabolism “slows”, and it doesn’t have a more efficient
“starvation mode” that uses fewer calories to perform the same tasks. (If it
did, natural selection would surely have dictated that we would always be in
that mode, and thereby need to eat less.) Rather, a “slowed” metabolism means
that you will either be slightly colder (unlikely to be by enough to make a
serious difference) or that you will have less energy available, and
therefore perform fewer tasks. Maybe you will feel like exercising less, or
maybe you will perform tasks in a lazier fashion, or maybe you’ll simply sleep
more. Putting all this together, it means that you can expect to lose some
fraction of your caloric deficit to temporary metabolic changes.
If you don’t get discouraged and continue dieting, your weight loss will
continue apace, but unfortunately you start to run into the problem discussed
above: your reduced size is resulting in a smaller energy expenditure. Research
suggests that for most reasonably large deficits, it would take years for
this effect to neutralize further weight loss (at which point you would have
achieved a new equilibrium), but in the mean time your losses each week will
get smaller and smaller. No wonder most people get discouraged! They end up
cheating or adding calories to their diets little by little, and eventually
they stall out. This is why many people hit plateaus in their diets. A slowing
rate of weight loss leads to discouragement, and discouragement leads to a
failure to adhere strictly to the diet, and this leads to you hitting a new
equilibrium point much earlier than expected.
Because of your body’s remarkable capacity to adapt to changes in food
availability, you have only two choices if you want to overcome this problem.
You can start out with a lower calorie budget, precisely calibrated to match
the metabolic changes you anticipate. Or you can adjust your calorie intake
as you go along, to maintain a constant deficit. I strongly suspect that the
latter approach is easier to follow and less discouraging, so it is the
approach PyWeight is based on.
Where is the obesity epidemic coming from?
This talk of equilibria makes an obesity epidemic rather surprising. Even if
people are eating more than before, why aren’t they simply reaching new
slightly higher equilibrium points, like our theoretical immigrant? After all,
if the way one developed a body that required 4000 calories just to maintain
itself was to start eating 4000 calories a day, no one would ever do that. Most
people with normal body weights couldn’t even stomach 4000 calories for many
But the evidence for epidemic obesity is clear. Here in the U.S., the state of
Mississippi had the highest rate of obesity in 1995, at 19.4%. In 2010, a mere
15 years later, Colorado had the lowest rate of obesity in the country, at
19.8%, and 12 states were over 30%.5 Today, more than 40% of American adults
have obesity.6 This is happening quickly enough that it doesn’t work to
posit genetic or behavioral changes in a new generation of adults: the same
people who were thin 10-20 years ago have obesity now. So what’s happening?
Of course, when we’re talking about the obesity problem, we’re talking about
averages. Some people do maintain an equilibrium point that’s just
overweight. Other people don’t become overweight at all. Yet in this country,
we seem to have reached the point where the majority of people put on weight
year over year, endlessly.
You can square an equilibrium view of body weight with this fact using the
suggestion that we’ve reached some kind of tipping point. For some reason,
people are driven to eat slightly more than they need to maintain, and for many
of them the rate at which they increase their intake has begun to outpace the
body’s compensatory mechanisms.
The body is calibrated to give you hunger signals that will result in you
eating approximately what you need to eat to maintain your current weight, on
the sort of diet most people had in the past. Someone who could maintain at
2000 calories a day and significantly overate7 would gain weight, but after
doing that their body would still only be giving them hunger signals
appropriate to maintain their new body weight — or more precisely slightly
less until they adapted. (Effectively, the two principles mentioned above are
functioning in reverse here.) So this person would not automatically continue
to gain weight. The body’s negative feedback of constantly feeling too full
would tend to keep their weight in a healthy range.
Our hunger signals are no longer accurate. The body still says “I need 2000
calories”, but it doesn’t feel full until you’ve eaten 2100, thanks to the low
satiety calorie-dense diets of today. When you gain enough weight on such a
diet, your body will start to say “I need 2100 calories”, but now it takes
2200 to fill you. And because eating food takes time and costs money, your
increased appetite will be drawn towards quicker and cheaper options, further
exacerbating the problem. Before, the body gave you you a signal that you
should eat less food the day after a big meal. Now, even a big meal isn’t
enough to make you feel full.
This suggests we should change our view of personal responsibility. People in
the last 30 years are facing a problem that virtually no one in the past faced:
their stomachs are lying to them about how much they should eat.
Recall the equilibrium equation: your equilibrium weight is a function of your
genetics, body composition, age, hormones, diet, and exercise. If you consider
this equation on a societal level, you can replace diet and exercise with just
one variable: environment. In the absence of a strong effort to control these
two variables, the health environment you are in will dictate them to you, and
therefore control your weight.
Faced with the environment that most people in Western countries faces, the
average person is going to gain weight over time. So on a broad social level,
that’s what we see. Blaming individual people for the consequences is not just
counterproductive, it’s an inaccurate accounting of what has happened.
Consider the following scientific model of average weight gain8:
What this model indicates is that the weight gain of the average American over
the last decades can be explained by a relatively tiny gap between energy
intake and energy expenditure. In fact, the average gap for the period examined
in this study was only about 7 calories a day. The problem doesn’t amount to
7 calories here or there, the epidemic stems from the fact that this gap is so
reliably sustained. High calorie junk food is causing many of us to overeat,
day after day, with no feedback telling us to cut back for a while. This has
gone on for so long that the average person (as of 2005) needed to cut out more
than 200 calories a day to achieve the equilibrium point of the average person
in the 1970s. This number, of course, has only continued to rise.
Watch this excellent five minute clip detailing some of the causes of calorie
overproduction in the United States:
You might be tempted to decide the takeaway of this analysis is political:
corporations are profiting off of human suffering on a massive scale, so some
social action is needed to fix the problem. Insert regulation, social
democracy, or whatever your preferred solution happens to be. That’s all well
and good, but it’s of little use to someone suffering the effects here and now.
This country will happily let you die in the street. In all probability, things
are not going to get better on a timescale that matters to you. The only option
is to take matters into your own hands and exploit the two variables under your
control to the maximum extent you can. For some people, that will be enough.
The good news
The one bright spot is that for most people the food environment is not yet bad
enough to make it impossible for them to control their weights. If you are not
very constrained in your diet and exercise choices, then you have a fairly good
chance of overcoming your environment.
As for the other variables in the equilibrium equation that are outside of your
control (e.g. genetics and metabolic effects), the good news is that these
almost always have a relatively minor impact on your weight. Most people have a
basal metabolic rate (the number of calories you burn each day just by being
alive) within 10 percent of the rate estimated by the Mifflin-St Jeor equation.
“Slow” metabolism is not usually the cause of weight gain.9
This means that many people can change their equilibrium point by managing
their calorie intake and becoming more physically active.
Addendum: it’s okay to try to lose weight, I think
Some authors, including philosopher Kate Manne, claim that encouraging
dieting and even dieting yourself are
I agree that negative self-image and exploitation in the diet industry are
both things we should condemn, but for this to have consequences for individual
dieting decisions something more is needed.
I think that’s why Manne’s article, though it defends an ethical conclusion,
ultimately rests on two empirical claims. She writes that “fat bodies can
be not only healthy but also athletic” and “the vast majority of diets fail to
make people any thinner or any healthier in the long term”. These claims are
both true, and suggestive, but don’t on their own deliver her conclusion.
It is true that some fat bodies are healthy. Similarly, many people who
don’t get the recommended 30 minutes of exercise every day are also healthy.
Despite this, since exercise and a reduction in overweight tend to result in
health improvements, it can make sense to pursue both.
Likewise, the fact that most diets fail cannot deliver the conclusion either.
If this claim is to show that it would be a bad idea for me to start a diet,
it needs to be true that I can have no expectation of success, so that trying
to lose weight is similar to playing the lottery. I think this is false. There
are reasons that most diets fail, but dieting is not tantamount to fighting
genetic or environmental determinism, as I argued in the previous section. I
explain further why I think it is reasonable to expect success in the next
I suspect that any versions of Manne’s empirical claims strong enough for her
conclusion will be false.
The claim that some fat bodies are healthy suggests a stronger version that
would suffice: most fat bodies are healthier than the same bodies would be if
they were thin. Dieting would then, in all likelihood, make you less healthy.
In this form, the route to Manne’s conclusion that encouraging dieting or
dieting yourself “is a morally bad practice” is obvious, but the premise seems
clearly false. There certainly is a correlation between body weight and health.
As I will discuss later, some (contentious) studies suggest that being
slightly overweight does not increase mortality, but no medical authorities
will endorse the notion that obesity is healthy.
Likewise, if diets could be shown to be inherently dangerous, unhealthy, or
doomed to fail, it would be easy to reach the conclusion. Manne hints at this:
she says that in one month early in her diet, she didn’t eat for 17 out of 30
days. This is indisputably a bad approach to dieting. If most people trying to
lose weight go for diet fads and ineffective products, or take dangerous
or disordered approaches like extreme fasting, that would help explain the
However, it seems false that these problems are inherent to dieting. I ate
food every day on my diet. Manne herself seems to have ended up with a
sensible diet: she lost about 50 pounds in a year, close to the common
recommendation of a pound per week. The diet industry is certainly blameworthy
for distracting people from methods likely to be successful and promoting
fads, but this alone can’t show that dieting is wrong.
How to succeed at dieting
Most diets fail. In a major longitudinal UK study, less than 20% of people with
obesity managed to decrease their BMI by at least one category without a
subsequent increase, over a 9 year period. 78% of people who lost at least 5%
of their body weight gained enough back that after 5 years they were back
within 5% of their original weight, or even higher.10 This indicates that
while plenty of diets do succeed (the popular claim that 95% of diets fail is
incorrect), successful dieters are by no means the majority.
I don’t believe a 20% success rate should be discounted. It suggests that diet
success is possible and that we should be looking at what the successful
people do. If we were to discover that a particular intervention to help people
quit smoking had a 80% recidivism rate, we would still encourage people to
take the program (because 20% is much better than zero).
Part of the issue with assessing the success of a typical diet is that if you
lower the bar of what counts as successful weight loss, then the success rate
goes up. If you demand studies involving weight loss of a quantity closer to
what I was hoping to lose, examples become even rarer. Most studies are testing
specific short-term interventions to see what works and what doesn’t, not
looking at how to achieve a long-lasting high level of weight loss. It’s hard
to get reliable data about the prevalence of successful diets resulting in long
term weight loss of 25% of body weight or more. A large portion of attempts
The reasons why most diets fail go beyond simple misunderstandings of what it
takes to succeed, of course. Food is tempting. Not eating food when you are
hungry sucks. The restrictions that a diet imposes can be difficult to
implement. The nutritional deficits imposed by Western eating can become even
more severe on a calorie restriction, especially for those who can’t afford to
make their own food. For many people, a personal trainer and gym membership are
unaffordable, and for some even a doctor visit is out of reach. (It was for me
when I began my diet! Thanks, America.)
Given the high rate of failure, what I went looking for was a diet that could
guarantee success to those who stick to it and and don’t cheat. My hope was
that following such a diet would not only bring me the weight loss I was
looking for, but that the certainty of success might provide additional
motivation to stick with the diet.
As a result of studying physics in college, I was naturally inclined to follow
a “calories in, calories out” diet. This comes with something like a success
guarantee: if you consume fewer calories than your body uses each day, you
will lose weight. Other diets may be successful as well, but only if they
end up resulting in calorie reduction anyway. Diets may have some
individualized metabolic effects but these appear to be minor in
comparison.1112 Perhaps some other diet is easier to maintain than
calorie restriction, but no other diet could provide the guarantee of
Counting calories has a reputation for being difficult, which is part of the
reason diets that dress up the calorie counting by using “points” systems are
so popular. It’s not only easy to underestimate the number of calories that
you’re eating, it’s also difficult to determine with accuracy how many calories
you’re burning. Moreover, if you weigh yourself once a week or so, your
measurements might get swamped by variations in water weight or having an
unusually large lunch, so it’s hard to know if you’re making progress. If you
weigh yourself once a day, the value on the scale is likely to fluctuate so
much that you constantly fly into a panic over an imaginary weight gain.
Furthermore, you might already be disheartened by reading the previous
discussion of why it’s hard to lose weight. As you lose weight, the number of
calories you need just to stay at maintenance is going to be decreasing. How is
it possible to reliably stay ahead of the curve? It turns out that there’s a
simple trick that can be summed up in a single paragraph, and this trick
obviates most of the complexity of weight loss. This trick is:
Record your weight every day. Every two weeks, calculate your approximate
weight loss using your record. If you are losing weight too quickly, eat
more the next two weeks. If you are losing weight too slowly, eat less the next
That’s it. Weight loss, summarized. By only worrying about whether your rate
of weight loss matches your predetermined goal, and not calories or points or
what have you, you can hone in on a reliable rate of weight loss over time with
relatively little effort. Accurately counting calories or determining your
daily expenditure is not required.
You can do this yourself with just pencil and paper if you like. But technology
will help you in two ways: you can more accurately calculate your rate of
weight loss using techniques like linear regression, and you can estimate how
many calories you missed your target by (instead of pounds), to make it
easier to adjust your intake in the future.
This idea was inspired by the book
The Hacker’s Diet. The book’s version of
this scheme goes something like this: if we assume that each pound of fat lost
contains about 3500 calories, then the math becomes quite simple. If you need
to lose x pounds, and you maintain a deficit of about y calories a day, it
will take you about 3500 * x / y days to do that. At a deficit of 500
calories a day, you will lose about a pound a week. If after two weeks you’ve
only lost 1 pound, you know you’re missing your target by 250 calories per day.
Unfortunately, this relies on the myth that a pound of fat contains 3500
calories. It turns out the correct number is closer to 4280.14 The slightly
better version of the theory, which says that the average caloric density of
the weight you lose is 3500 calories, is an inaccurate approximation. You do
lose some fat free mass while dieting, and this contains around 824 calories
per pound. So 3500 calories per pound is only accurate if about 3/4 of the
pounds you lose are from fat. This is only a reasonable estimate for very
overweight people at the beginning of a diet. Later on, as you approach an
optimal body weight, you will lose a larger proportion of lean mass. If you
were to continue losing weight indefinitely, you would of course eventually
have nothing to lose but lean mass. This would be extremely bad for your
Because of this complication, PyWeight uses an equation from a real diet
model15 to estimate how many calories were contained in the weight you lost
in a given interval, and therefore how many calories you are away from hitting
your weight loss rate target. This means it’s possible to target a linear rate
of weight loss despite all the non-linear effects. (For what it’s worth, when
you’re targeting a constant rate of weight loss rather than a constant,
pre-determined calorie budget, the problems caused by inaccurate estimates are
greatly lessened. In fact, you’d probably be fine with the old 3500 calories
per pound estimate. The upshot of PyWeight’s advice is usually just “try to
eat a little less / more next week”, just as it would be if you were using
pencil and paper.)
With the Hacker’s Diet, the calculations are all done in a series of
complicated spreadsheets which are provided with the book. PyWeight
allows entering your measurements directly in a built-in log, and all
calculations are done for you automatically. It also provides intake adjustment
advice at a user-selectable frequency.
The linear regression method used by the Hacker’s Diet can also be improved
upon. Doing a naive linear regression by breaking the data into intervals
lasting 2-3 weeks means that only the data from those weeks figures into the
calculation of your current rate of weight loss. But this is throwing away
data; you can determine a much more accurate starting weight for each interval
if you also make use of the data from the previous interval. PyWeight takes
exactly this approach with a technique called spline interpolation. A possible
alternative would be to use a moving average, but I believe this would give
worse results because of the high degree of variability in weight measurements
and the fact that the diet is built around attempting to lose weight at a
As a side effect of this improvement, PyWeight is able to generate very nice
looking graphs of your weight loss. Here is mine from the last ~7 months. (I
lost the first 20 pounds using the methods described in this article before I
The idea behind using a program to track your weight is that despite the fact
that our hunger systems are broken because of modern calorie dense diets, we
can use technology to replace this body function, letting simple math
regulate how much we eat. This method actually works even better than simple
calorie counting, because it adapts to your changing energy needs as you lose
weight. If you want to keep up a one pound per week deficit, you’ll need to
reduce your intake over time, and this approach will help you do so
Even though you’re measuring your weight every day, you don’t have to worry
about what the scale says in the way you do with ordinary calorie counting. An
unusually large meal might send your apparent weight up 1-3 pounds for several
days, but this is mostly water weight that will soon go away. Looking at a
statistical measure of your weight loss over a period of weeks smooths out
these misleading bumps in the graph.
Looking at my graph, you can see several of these blips, some of them quite
large. My daily scale measurement increased 6.8 pounds over the course of four
days in late December, but this did not represent any long term weight gain.
I’m not going to tell anyone that this approach makes dieting a sure thing.
A great deal of willpower is still involved. But knowing exactly what you’re
doing and trying to keep up a consistent rate of weight loss makes it much
easier. Steven Novella helpfully
I would conclude that weight control is difficult, but not impossible. In
addition, I believe that the popular weight loss industry is making the problem
worse by distracting the public from those strategies for which there is at
least some evidence of efficacy and focusing on minute, short term, and
probably insignificant effects from manipulating macronutrient proportions.
My goal with this diet is to focus on what we know works (consistently eating
fewer calories than you burn) and find a way to make sure I am doing that with
small course corrections over time. I began this diet with a high degree of
confidence that it would work if I didn’t get burned out and give up, and that
confidence was rewarded. As you can see from the graph above, I was able to
maintain an extremely consistent rate of weight loss over time; I never
deviated from my monthly target by more than a few percent (except around the
holidays and a strenuous four day hiking trip in mid-February during which I
snacked almost continuously).
It’s not just losing the weight that gives me confidence this method works.
It’s the fact that I lost the weight at almost exactly the rate I planned in
advance! I think many individuals can see success in dieting if they can follow
a similar approach.16
No method for individual correction can solve the underlying problem. So long
as the United States fails to get a handle on its poor nutrition environment,
obesity will continue to be a serious issue. Proposing diets as the solution
is like thinking that encouraging people to recycle can prevent climate change.
However, until the social causes of obesity are resolved, individuals are on
their own. Diets will not reverse the long term obesity trend, but they can
help dedicated individuals who are fortunate enough in their circumstances.
I started out just above 240 pounds, which at my height was a hair into the
obese range (30+) by the BMI metric. BMI is not necessarily a good indicator of
individual health, but the truth is I felt bad. I did not feel like
exercising, or even going for a walk. Hiking long distances, one of the
activities I’ve enjoyed the most over the years, was certainly out of the
question. While it’s not impossible to be in good shape at the size I was,
it’s certainly much more difficult.
How bad is it, in general, to be overweight? There has been some recent
controversy on this point in the medical world. The long-time consensus has
been that while individuals with BMIs between 25 and 30 are not necessarily
unhealthy, overweight is correlated with heart disease and diabetes, as well as
increased all-cause mortality.
One 2005 study suggested that being in the overweight category was associated
with reduced all-cause mortality relative to the normal (18.5 - 25) BMI
category.17 A follow up meta-analysis18 by the same primary author
confirmed this finding, and also found the difference between mild obesity and
normal body weight to be insignificant. However, a much larger third
analysis19 disagreed, finding that after controlling for smoking, prior
chronic disease, and length of follow-up, both overweight and obesity were
correlated with increased all-cause mortality risk. Moreover, they were able to
reproduce the findings of the second paper by removing those quality controls.
(The danger of leaving in smokers is that a disproportionate number of smokers
will have healthy body weights because nicotine functions as an appetite
suppressant. Smoking is much worse for you than a few extra pounds, so these
people make “normal” body weights look less healthy on average.)
I’m not here to say that the third study was right and the first and second
were wrong. I don’t have the expertise to do that, to begin with. The primary
author of the first two papers wrote a response paper which claimed that the
approach taken by the third was biased and suggested that the exclusion of
smokers might lead to more bias than leaving them in.20 The result has
been an increasingly bitter academic feud over claims of “bias” with no
obvious conclusion (to me).
However, a very nice fourth paper21, a systematic review of the issue,
reached more or less the same conclusions as the third. It was significantly
larger than either of the others, containing almost 10 million individuals
in the never-smoking category alone, and none of the authors overlapped with
those of the third study. This study found that an ideal BMI was approximately
23, with negligible differences between BMIs from 22 to 25. Furthermore, the
paper reproduced the most important finding of the third: the conclusion was
reversed when filtering was removed. A BMI of 27.5 was associated with a lower
mortality risk than one of 23, and a BMI of 32.5 lower than one of 20!
Because this study subdivided “normal” BMI into many categories, the
conclusions are a bit more fine-grained. The results may help explain why in
some studies the 18.5-25 BMI range compares unfavorably with the 25-30 range.
The dangers of being underweight are quite extreme: among never-smokers, the
study found that a BMI of 17.5 was associated with a 35% increased mortality
risk. Someone with a BMI of less than 20 contributes to the average outcome
for the normal BMI range (if their BMI is 18.5 or higher), but they’re exposed
to more of the risks of underweight compared to someone with a BMI of 23, which
the study found to be optimal.
I believe it’s very important for casual interpreters of medical research to
remember to look at papers in context. All this bickering is essentially over
a 5-10% difference. If you believe the fourth paper, a BMI of 27.5 is
associated with a 7% increased all-cause mortality risk versus a BMI of 23. If
you believe the second paper, having a normal BMI might be associated with 6%
more all-cause mortality risk versus an overweight BMI. Personally, I’m
inclined to believe the former, because the studies are larger, it’s more in
line with the medical consensus view, and because the exclusion of smokers in
the analysis seems justified. But I see no reason for me to argue with
someone who believes the latter. To my mind, the sane middle ground is found by
keeping your BMI well out of the obese range, and at a level where you feel
able to keep yourself in good shape. My guess is that the difference between
the two is swamped by just getting plenty of exercise.
For me, the truth is that I wasn’t able to feel healthy or stay fit at the
weight I was, and the BMI number merely served as a painful (though possibly
useful) reminder of that.
For the sake of having a goal weight, I picked the arbitrary target of 160
pounds. This happens to correspond nicely to taking 10 points of my BMI, and
struck me as unlikely to be unsafely low. It corresponds to a BMI of slightly
above 20, and my plan is to put on a few pounds of muscle over the next year.
The last time in my life when I was thin was around the age of 10 or 11 (when I
had temporarily become a vegetarian), and I wanted to experience that again.
It remained only to choose a rate at which I was going to try losing the
weight. The UK’s National Health Service
a weight loss rate of 0.5 to 1 kg a week is safe. 1 kg a week is equivalent to
9.45 pounds lost every 30 days. I rounded this up to a target of 10 pounds per
30 days. I’m considerably taller than the average person, and so there is a
larger gap between my daily energy expenditure and my basal metabolic rate than
for the typical dieter. I think this was a safe target for me.
I emphasize that if you’re going to pursue fast weight loss, you should talk to
a doctor about that. You may very well have more success and invite fewer risks
by targeting a pound per week (for some people, even less). In my case, without
access to professional advice, I made the best judgment I could about what
would be safe and effective for me.
How to get enough nutrition on a diet that doesn’t suck
It can be helpful to get a rough estimation of how many calories you should
budget for yourself. In my case, something like 1800 calories a day at the
start of the diet was appropriate. One nice thing about taking a statistical
approach to weight loss is that even if you screw up this estimate quite badly,
you’re unlikely to put yourself in any danger as long as your target rate is
reasonable. If you lose too much weight in the first weeks of the diet, you
will receive a sharp corrective directing you to eat several hundred more
calories every day. PyWeight is thus self-correcting.22
Cutting a thousand calories out of your usual intake can make it difficult to
get the nutrients your body expects to have available. It’s important to have
a plan for getting the necessary vitamins, minerals, and essential amino acids.
This is an area where The Hacker’s Diet is not very helpful. It says
Since we’re efficient food processing machines, it’s possible to reduce all
the complexity of food to a single number that gives the total energy the body
can extract from it—the calorie.
The extreme form of this is the infamous Twinkie diet. You
will lose weight on such a diet; it might also kill you. (It’s interesting that
the actual Twinkie diet guy didn’t just eat Twinkies. A full third of his
calories came from fruits and vegetables along with a protein shake. Eating no
protein at all for months on end would probably not have gone well.)
To be fair, the book also suggests eating a balanced diet. As a vegan, I think
there’s some cause for me to be concerned, especially on a diet as restricted
as mine. I took a vegan multivitamin (DEVA
tiny) every day of the diet, to ensure I wasn’t vitamin deficient. I really
like this multivitamin because (a) it provides ~100% RDA of vitamins I was
likely to be deficient on, instead of 10000% like some multis with potential
side effects, and (b) it was small and easy to swallow. I also have a B-12
supplement that I take about once a week (as do most strict vegans).
So the remaining concern was protein. Getting enough protein isn’t generally a
problem for vegans23 and even on a restricted diet it wasn’t one for me
either, at least in terms of quantity. The issue is that there are
minimum amounts of
specific amino acids that we have to consume for building cells and other
important stuff our bodies do. I wanted to make sure I was hitting the WHO
minimum recommendations fairly consistently, even though I didn’t think a mild
deficiency for a few months was likely to be harmful. (After all, the Twinkie
guy didn’t fall apart.)
In order to hit these targets, I ended up on a diet that was pretty protein
loaded. I hit the recommended amino acid intakes almost every day, even though
I was on a diet with considerably fewer calories. At the same time,
I also emphasized eating a lot of fiber and low-density foods. The point of
this is that both fiber and foods with a high bulk to calorie ratio will make
you feel full. Note that neither of these strategies will directly cause you to
lose more weight than otherwise. They’re simply ways to make a low calorie diet
easier to maintain by reducing hunger.24
Many vegan foods are complete proteins (that is, they include all of the
essential amino acids). Most aren’t perfectly balanced between the amino
acids, so you might have to eat more than enough of one to get enough of all
the others. (This is one of the reasons why a balanced diet is often suggested,
since eating many different protein sources will help balance the ratios out.)
Note that this problem isn’t unique to plant-based diets — beef isn’t perfectly
balanced either, nor are most other animal-foods. It’s only something to worry
about for someone on an intense calorie restriction.
In trying to come up with meal plans, I found WolframAlpha indispensable. You can enter an amount of food,
and it will tell you how much of each amino acid it has (assuming it’s in their
Lentils and soy turned out to be useful for hitting my high protein and fiber
requirements on a low calorie diet. It’s actually possible to hit the WHO amino
acid recommendations and much more than 100% of your fiber using mostly these
sources on a mere 1200 calories a day.
You don’t have to eat the same thing constantly on this diet. (In fact the
whole point is that you don’t need to be precise about what you eat or count
calories exactly.) However, if you’re cutting so much that you need to be
concerned about nutritional deficiencies, planning your meals may help. It’s
probably still best to mix it up some of the time, just to ward off any
additional danger of malnutrition.
On many days, I ate whatever kind of food I wanted (or even went out to eat)
and just went with a rough estimate of my calories. The point is that as long
as you are hitting your nutritional requirements, and can get even a rough
estimate of the number of calories you eat, you’ll be fine. The best part is
that you don’t even have to write anything down if you don’t want. It’s a
calorie “counting” diet, but all the counting is quietly being done
automatically for you by enhancing the quality of your estimations over time
until they’re about as good as a real count. (Of course, should you struggle
making the necessary adjustments to hit your targets, you might find a food
diary indispensable, at least for a short time.)
If it is at all feasible for you, I do suggest making as many of your own meals
as possible, and making them as highly nutritious as you can. Fast food is
far worse than you or I probably realize, and it tends to be extremely calorie
dense without satisfying.
Keys to success
I think adopting the following attitudes and practices while dieting helped me
Commit to a lifestyle change, not a brief period of restriction.
As discussed above, this is actually mandatory if you want to keep the
weight off afterwards. This means that you need to recognize that you’ll be
giving something up (you will have to permanently eat less food), and
decide that the cost is worth it.
Commit to weighing yourself every day.
Forming any habit can be irritating, but when your body has been sending
you dishonest signals for years, this is really the only choice. Many
people who are successful at maintaining weight loss continue to weigh
themselves regularly for years afterwards.25 You’ll need to do so at
least until your hunger system is retrained.
Don’t declare any food off limits (other than pre-existing dietary
There’s no good reason to set yourself up for a cycle of failure and
self-blame. One of the most common reasons for dieting failure is that
people prohibit certain foods entirely, crave them incessantly, and
then binge when the pressure proves too much or they have a “cheat” day.
The resulting feelings of shame may drive them away from the diet entirely,
especially if the number on the scale goes up the next day. (Usually, this
is a temporary result of eating a lot of food. The whole point of using a
statistical measure of your weight loss rate is that you don’t have to
worry about the daily fluctuations!) Throw out the concept of cheat days;
every day is a diet day, but every day is also a day that you may have a
doughnut if you absolutely must have one. As long as the line on the weight
chart keeps going down at the rate it’s supposed to, you have nothing to
Try to develop a more healthy decision making process about what you eat.
Ask yourself: am I willing to trade being hungry later for eating this
sugary thing now? Speaking for myself, I ate almost no added sugar during
the diet. It was never prohibited, though. I put a teaspoon of real
sweetener in my tea. I had a little square of a chocolate bar once every
week or two. I had a slice of pie or a few cookies over Thanksgiving and
Christmas. I had a beer every now and then. Nothing off limits.
Never give up.
There will be days when you want to. If you have a tendency towards anxiety
or depression, or you use food to cope with problems, you are going to
have to face that head on. You’ll have to handle feelings of hunger and
reduced energy. It’s even likely that there will be days where you screw it
up. The difference between the people who succeed and those who fail is
that people who succeed don’t tell themselves that they failed. Failure
isn’t what happens when you have one bad day, it’s what happens when you
I lost more than a third of my original body weight. The guarantee that you
will lose the weight if you follow the simple instructions output by the
program and dutifully weigh yourself every day was enough to push me through
the hunger pangs and inclination to give up. Here are the usual before and
Is this a good way to lose weight? For me, the dominant consideration in
answering this question is “is this a successful way to lose weight?”. That’s
because most diets are unsuccessful, and so successfully losing a significant
amount of weight is already a high bar to clear.
I think it’s reasonable to say that this can be a safe way to lose weight.
If losing the standard recommendation of 1-2 pounds a week is a safe rate of
weight loss for you, PyWeight will not tell you to do anything stupid. It is
merely a tool to help keep you on track. You can choose a slower rate of weight
loss and it will come off just as reliably, but will require more patience.
The other question, of course, is can I keep it off? That remains to be seen,
but I’m planning to continue weight tracking for the rest of my life while
eating at maintenance (net-zero calories in-out every month). This is something
PyWeight can also help you do. If you set your desired deficit to zero, any
time you begin to gain weight PyWeight will quickly warn you to reduce your
The fact of the matter is that even if I knew for certain that I’ll gain it all
back within five years, I’d still do it all again. I really do feel that much
better, just as a result of the weight loss. You can’t assess the quality of
someone’s life by what they happen to weigh when they die; even if I eventually
regain weight I will have bought myself years of enjoyment in the mean time
that can’t be taken away. That said, I understand that most people are looking
for long term weight loss. I hereby commit to publish an update to this article
with my current weight one year from the date this was posted, and then again
a year after that.
Thanks to anyone who took the time to read all the way through this. Feel free
to ask me any questions that you have.
PyWeight is free and licensed under the GPL v3, which means you can
modify it under certain conditions to meet your own needs. ↩
Hall, K. D., Heymsfield, S. B., Kemnitz, J. W., Klein, S., Schoeller, D.
A., & Speakman, J. R. (2012). Energy balance and its components: implications
for body weight regulation. The American journal of clinical nutrition, 95(4),
One failing of the NIDDK model referenced in  is that it doesn’t seem
to show this when modeling long term maintenance. When you put in a starting
weight of 235 pounds, the model says you would require 3050 calories to
maintain. For someone who gains 85 pounds to reach 235, it says they will
maintain at 3400. No matter how long you run the simulation, it’s hard to get it
to drop this maintenance level significantly below 3400. This is logically
impossible. The only way someone becomes the first sort of person who
maintains at 3050 is to first be a person who weighs 150 pounds and gains 85. It
must be possible to transition from one to the other, although their model
does not show this. Moreover, if we assume that your metabolism does
eventually adapt, a 100 calorie decrease in your intake should eventually
result in a loss of closer to 22 pounds. ↩
This sort of sporadic overeating could actually be a helpful adaptation.
If you live in a place where food is often scarce, a rare occasion when a fatty
or sugary food was available might be a good time to gorge yourself well past
the point of satiation, in anticipation of lean months to come. ↩
Hall, K. D., Sacks, G., Chandramohan, D., Chow, C. C., Wang, Y. C.,
Gortmaker, S. L., & Swinburn, B. A. (2011). Quantification of the effect of
energy imbalance on bodyweight. Lancet (London, England), 378(9793), 826–837.
Fildes, A., Charlton, J., Rudisill, C., Littlejohns, P., Prevost, A. T.,
& Gulliford, M. C. (2015). Probability of an Obese Person Attaining Normal Body
Weight: Cohort Study Using Electronic Health Records. American journal of public
health, 105(9), e54–e59. https://pubmed.ncbi.nlm.nih.gov/26180980/↩
For example, much has been made of the supposed ability of ketogenic
diets to increase metabolism. https://pubmed.ncbi.nlm.nih.gov/16389240/
discusses this and attributes to a low-carbohydrate diet that resulted in
ketosis an increased expenditure of less than 100 calories a day. (This was
attributed to increased thermogenesis, not any special property of ketosis.)
It’s not nothing, but it indicates that most of the success of a ketogenic diet
will be from calorie reduction anyway, along with potentially increased satiety.
A high protein diet could also potentially provide a feeling of having more
energy, resulting in an increase in physical activity. In my diet, I decided to
eat plenty of clean protein for exactly these reasons, along with the desire to
achieve a complete amino acid profile. I think these reasons make sense, even
though I’m not counting on “ketosis” as a weight loss mechanism. ↩
Given footnote , you might expect a typically high-carb vegan diet
to show the opposite effect (reduced metabolism). Not so. On the contrary, a
study looking at the effects of a vegan diet found that it increased the
thermic effect of food by 16%, which was associated with a significant increase
in weight loss compared to a control group.
https://pubmed.ncbi.nlm.nih.gov/31021710/ Despite following a vegan diet
myself, I don’t believe the overall results for this or any other diet show
that you can “hack” the weight loss process by causing the body to run hotter,
at least not significantly. In reading many papers while writing this, what
struck me the most was how many of them were limited or flawed outright. I
believe it is best to follow a diet that is healthy and sustainable over the
long term, even if a few clinical trials would have you believe that one
approach or another could net you a 10-20% faster rate of weight loss. On this
note, one very large study found that following a vegetarian or high-carb diet
was positively correlated with possessing a lower BMI.
https://pubmed.ncbi.nlm.nih.gov/11320946/ Thus, regardless of its weight loss
credentials, a high-protein diet may not be as successful at weight
A calorie-restriction diet can only guarantee success if you actually
follow it, which means accurately calculating your energy expenditure (the
number of calories you burn every day) and accurately counting the number of
calories you eat. Both are hard to do. My methodology gives you a
quasi-guarantee of success if you can follow instructions like “eat fewer
calories this week than last week”, even if you don’t know your true metabolic
rate or can’t count calories accurately. ↩
I mean this in the sense that well-situated people can have a high
degree of confidence in their success. Luck is involved in the question of
whether you are “well-situated” or not. Many people are not so fortunate: if
you don’t have time to cook your own food, are in a “food desert”, have limited
access to health care, are the victim of misinformation, or have mental health
issues or an eating disorder, then you’re unlucky in the latter sense. The
unfortunate fact of America’s public health crisis is that most people are in
at least one of these groups. That’s why public health experts don’t see
individual behavioral changes (dieting) as the solution to the problem. It’s
not that relatively well-placed, high-information people can’t see success (if
they approach dieting with a reasonable plan), it’s that most people don’t have
that access. So when I say that dieters do have some control over whether
they succeed, this doesn’t mean that a dieter who fails has only themselves to
blame. Individuals are “free” to quit eating as much in much the same way as
you are “free” to quit your job. It’s true, but it misses the point. ↩
Flegal, K. M., Kit, B. K., Orpana, H., & Graubard, B. I. (2013).
Association of all-cause mortality with overweight and obesity using standard
body mass index categories: a systematic review and meta-analysis. JAMA,
309(1), 71–82. https://pubmed.ncbi.nlm.nih.gov/23280227/↩
Global BMI Mortality Collaboration, Di Angelantonio E, Bhupathiraju ShN,
t al. Body-mass index and all-cause mortality: individual-participant-data
meta-analysis of 239 prospective studies in four continents. Lancet.
Aune, D., Sen, A., Prasad, M., Norat, T., Janszky, I., Tonstad,
S.,Romundstad, P., & Vatten, L. J. (2016). BMI and all cause mortality:
systematic review and non-linear dose-response meta-analysis of 230 cohort
studies with 3.74 million deaths among 30.3 million participants. BMJ (Clinical
research ed.), 353, i2156. https://pubmed.ncbi.nlm.nih.gov/27146380/↩
Within reason, of course. If you modify the source code of PyWeight
to let you target 5 pounds of weight loss per week, then the program will
happily tell you that you need to eat a negative number of calories.
Statistics can’t force you to start out with sane parameters, but as long as
they are reasonable for someone of your height and weight (e.g. 1-2 pounds a
week for most people), PyWeight will guide you onto the proper path. ↩
Craig, W. J., Mangels, A. R., & American Dietetic Association (2009).
Position of the American Dietetic Association: vegetarian diets. Journal of the
American Dietetic Association, 109(7), 1266–1282.
Goodreads is a popular site for tracking
books you have read and finding new ones using your ratings. Many people
have noticed that Goodreads ratings have a surprisingly small variance:
it seems like most popular books have ratings around ~3.8. This suggests
that a book the community assesses as fairly poor and one that it says
is great will be closer in rating than the descriminatory capacity of a
single user (you can’t grant half-stars in your Goodreads ratings)!
Is the situation really as bad as all that? Can anything be done about it?
One user scraped Goodreads’ API fairly recently and made the data
I downloaded this data and performed some analysis on the distribution
of ratings on the site.
One thing I found right away is that “only” about half of the films in the
dataset have ratings between 3 and 4. There’s a bigger issue, though: about
10% of the films have a rating of precisely 4. That’s because this includes
a ton of films with only a few ratings, and 4 is the most common rating on
the site. What we need is a model.
What we’d like to have is a distribution function describing our knowledge
about the dataset. This is usually called a Bayesian prior. Let’s consider
a simplified example.
If the data were in the form of thumb-ups and thumb-downs, our data would
consist of a proportion of thumb-ups out of the total number of ratings. In
other words, it would be
binomial data. This
entails that our prior distribution would be the corresponding
conjugate prior, the Beta
distribution. If we fit a Beta distribution to the data, we would get a
function with two parameters, α and β.
These parameters have a deeper meaning. Bayesian update allows you to
introduce new information to the prior knowledge you have, and tells you
what your resulting beliefs about the question at hand should be. When your
prior is a Beta distribution, the parameters are simply added to the new
evidence. If you have a new book with x thumb-ups and y thumb-downs,
the prior evidence suggests that the book’s true proportion of thumb-ups to
thumb-downs is x+α : y+β. Because the resulting rating is adjusted by
adding these “fake” ratings, α and β are sometimes called
Alternatively, with a simple substitution, you can think of yourself as
adding c = α + β fake ratings with the average value m = α / (α+β).
This approach is given the name “bayesian average” by
Paul Masurel, here.
We don’t have just two possibilities, of course, we have five: five
different star ratings. A distribution with several options seems
like it would be appropriately characterized by the generalization of
the beta distribution known as the
This approach is what Masurel suggests:
If you have the relevant data and infinite time, you may set these two
values by fitting a Dirichlet distribution on the dataset of the ratings of
all your computer books. However it is very common to just choose a pair of
parameter that mimick the behavior that we are looking for.
I’ve seen this approcah recommended by several others as well.
Okay, I have the relevant data and infinite time, but let’s consider
whether fitting a Dirichlet really makes sense here. I believe that it
actually does not!
The problem is that the Dirichlet (and actually, the beta distribution)
are conjugate priors for distributions of categorial variables, not
discrete ones. The binomial distribution, for instance, has success
and failure conditions, not 0 and 1. The multinomial distribution that
has any number of distinct categories.
Why is this an issue? Well, suppose we had distinct categories. Users could
describe their reaction to a book as “green”, “orange”, “red”, “yellow”, or
“blue”. (Assume that users wouldn’t immediately derive a mutually agreed upon
ranking system for these colors, i.e. they really are just colors.) In that
case, there would be no inherent correlation between colors, and you couldn’t
average them into an overall color. Star ratings are precisely the opposite.
They’re numerical ratings giving a quality score, and usually what we care
about is just the average.
If you fit a Dirichlet distribution to star ratings, the resulting distribution
will be underconfident about the behavior of the average star rating,
because it treats the ratings not as comparable numbers but as incomparable
categories. More specifically, a set of star ratings will converge to their
individual true values as new ratings come in more slowly than their average
will converge to its true value.
As proof of this, here is the result I get when fitting the
Dirichlet distribution to the Goodreads data.1 I am plotting the Dirichlet
reparameterized to show the strength of the prior on the average value.
As you can clearly see, the result is underconfident. Don’t use the Dirichlet
as a prior for star ratings, especially when you only care about the average!
What should you use instead? I think the beta distribution is actually
more reasonable here. Instead of modeling the proportion of successes and
failures in a series of Bernoulli trials, we can think of it as modeling the
average value of a series of individual equally weighted fractional
successes, where e.g. 5 stars means perfect success and 1 means complete
If anyone knows of a better way of motivating the use of a particular
distribution here, please do contact me by email or on Github and let me know.
I filtered out books with fewer than 50 ratings, to eliminate books for which
no rating consensus has yet been formed. (I also implemented some special
handling for books that have at least one star rating which no one has yet
given them, though this is a technical detail which I’ll let the interested
read the code to see.)
The beta distribution appears to be a very good fit for the data.
Experimentally, the distribution of dirichlet averages seems to become more and
more beta-like as the strength of the prior increases.
The smoothed and filtered values show the Goodreads ratings to be worse
(less informative) than we initially feared. Fully 80% of adjusted
ratings are between 3.5 and 4.2, a range considerably less than the one
star’s worth of discrimination available to individual users when rating
a book. Virtually every book on the platform that has enough ratings to
be reliable gets at least a 3, and virtually none gets more than a 4.5.
A partial solution
Given that we have a model of the data, we can adjust the average rating
of books with too few ratings to give ourselves a more accurate estimate of
the true average of the book. Furthermore, we can use the model to determine
what percentile a given rating corresponds to: my analysis suggests that a
book with a Bayesian rating of 4 is better rated than 2/3 of books on
For those who are interested in more accurate ratings (for books with a small
number of them) and the ability to see a book’s percentile, I have made a
which does just that!
Does the ability to see rankings instead of a raw score solve the problem with
Goodreads? It certainly makes things better, but in my opinion it doesn’t solve
the basic problem, which is that Goodread ratings just kinda suck. They’re
inconsistent, prone to audience biases, and fairly arbitrary.
As it happens, I think all of these works are great. (Books I hated are
similarly all over the place.) The problem is not just that the ratings
are so widely varied, it’s that there’s no apparent rhyme or reason to the
order these books are in. When a film I think is great has a relatively low
rating on IMDb (example:
I’m Thinking of Ending Things), there
is usually an obvious, comprehensible reason why most people didn’t like (or
were indifferent to) the film. Likewise, I thought
Star Trek Into Darkness was quite bad,
but I understand why most people liked it. Nothing of the sort can be said for
Goodreads: not many people dislike Shakespeare — so what’s happening here?
Unlike with IMDb, where the ratings (though clearly inflated, with most movies
between a six and an eight) do frequently indicate something about the quality
of the movie, I’ve found no way to extract the same value from ratings on
Here, I’m eliding how we arrive at the precise α and β values: the
Dirichlet is parametized by a vector α that behaves exactly the same way as
the α and β values of the beta distribution. Since our data has 5
possibilities, α will contain 5 values. To update a Dirichlet prior, each of
the values is added to the corresponding value in the evidence vector. As
Masurel shows, since we care only about the average value we can simplify the
resulting α vector down to two variables; the sum of the vector, and the
average (number of stars) that a book with exactly α reviews would have.
These values are just the c and m of the bayesian average approach. ↩
Wordle is a simple online word game.
It is played by similar rules to the classic game
Mastermind, which is
known in the United States through the version licensed to Hasbro. Wordle is
played with words instead of arbitrary configurations of pegs or numbers, and
there is one added twist: each of your guesses must also be a word in
Wordle’s dictionary, not any arbitrary configuration of letters.
For those unfamiliar with Mastermind, I recommend clicking through to try the
game or reading this brief description of the rules: every day, the game
randomly chooses a five letter word from a predetermined list of possible
words. On each of your turns, you will guess a word that must be in a
dictionary contained by the game. If you guess the word within 6 turns, you
win. The game tracks your streak of wins and the number of guesses it takes
you to solve each one.
For each guess, the game will tell you whether each letter
does not exist in the correct answer at all
exists in the correct answer, but is at the incorrect location
is the correct letter and is at the correct location
Each letter in the guess has a 1-1 correspondence with the same letter in
the answer, if there is one. This means that if the correct answer is “means”,
and you guess “green”, you would see the following:
The yellow color indicates that the chosen letter appears in the solution, but
that it is in the wrong place in our guess. The second “e” appears in grey,
because there is no second “e” in the solution. There must be a 1-1
correspondence between each guessed letter and a solution letter, and if they
were both highlighted in yellow, the “e” in the solution would correspond to
two letters in the guess.
Likewise, a green letter, indicating a letter in the correct location, takes
precedence over a letter in the incorrect location. So if the solution was
“rules”, we would see the following:
Here as before only one “e” is highlighted, but the second is in green rather
than the first being in yellow, since the letter in a correct location takes
precedence over a letter in an incorrect location.
I’d like to solve the puzzle
When I come across a puzzle like this, I’m immediately compelled to think about
optimal ways to play it. For example, the player is in the same knowledge
position at the beginning of the game every time. Since the opponent is not
adversarial (a random 5 letter word is picked, not a word intended to be
problematic for any particular guessing strategy), the optimal strategy
necessarily means guessing the same word first every time. Few human players
likely play this way.
In fact, given that the solution word is drawn from an unchanging list of words
before the game begins, this means you can just pre-calculate the optimal guess
given each previous guess and response from the game, laying out every possible
game state in a tree-shaped diagram.
I have written a solver for Wordle in Python which searches for an optimal (per
several rules, see below) game tree, and saves it in a JSON file, which turns
out to only contain about 15 KB when compressed. The optimal game tree for
Wordle therefore turns out to be pretty small!
How the solver works
The ideal game tree would have 3 constraints:
Every guess is a word that the game will accept
The maximum path to a solution is less than seven guesses
The average path to a solution is minimized
I have made several simplifications in order to quickly get a reasonable
Every guess is a word that the game will consider using as a solution
Each guess results in the smallest number of live possible solutions, on
A constraint to keep the maximum path under seven guesses turns out not to be
necessary, because a solver with just the two constraints above will never take
more than five.
As it turns out, the game actually has two word lists: one is the set of words
it will consider using for solutions, just over 2000 words. The other is the
full dictionary of words it will accept as guesses. I found both these lists
guessable words would in same cases allow faster solves, but my solver is so
efficient even without this that I haven’t seen fit to implement it yet. So my
program only guesses words that could, in theory, be used by the program as
Likewise, always guessing the word that will result in (on average) the
smallest number of possible solutions is only an approximation of optimal
guessing. There are 2314^6 = 153525361154699100736 different possible routes
through the game, and so brute forcing your way to an optimal solution is
(while not unthinkable as it is in chess), probably unworkable in Python.
Clever pruning of the search tree should help (and I’m going to look at this at
some point), but is not as easy as it is in games with an opponent. In chess,
you can simplify the tree by assuming that the opponent will always make the
move that is worst for you. In Wordle, because the solution is random, you
have to always consider all possibilities and try to find the guess that is
optimal on average.
Still, though, reducing the average number of live solutions as much as
possible is a very good approximation of ideal play. Consider the following
After the second guess, the computer makes the interesting choice “pygmy”.
This may seem counter-intuitive. Not only do we already have two vowels
confirmed to be in the word, “i” and “e”, the guess actually contains “y”
twice, which reduces the number of letters contained in the guess. As a matter
of fact however, this guess is rather astute. The computer will always find
the solution on the fourth guess depending on the outcome of this query.
Using ‘A’ to mean absent, ‘P’ to mean present, and ‘C’ to mean correct:
Being in a point in the game tree where there are six descendent guesses
means that there are six possible solutions. Clearly, optimal behavior would be
to always find the solution on the next turn. A brute force search would find
this solution, but so would the heuristic of eliminating as many possibilities
as possible. With “pygmy”, we always eliminate five possible solutions, the
best result possible in this case.
Using a heuristic like this is much faster than a brute force search, because
it generates a guess in each situation without needing any recursion at all.
This was just a quick little project for me, taking a few hours, so the code is
relatively unoptimized. Determining the best first guess requires searching
every combination of possible guess and possible solution, and this takes
several hours. The complete tree is generated in only a few minutes after that.
Because it is stored as JSON, a player for the game is included that doesn’t
have to do any searching - it simply reads the next guess out of the game tree.
The solution is found, on average, in 3.51 guesses. In the worst case scenario,
the solution is found is 5 guesses. The histogram of outcomes is as follows:
If anyone improves on my solution by utilizing the complete dictionary or
achieving a brute force solve of the game, I would be curious to hear how much
you manage to improve on these statistics.
Jan 11: I discovered that the program was treating all guesses that resulted in
the same average reduction to the live possibilities as equivalent. We can,
without violating the constraints given above, choose to prefer guesses that
are also possible solutions to the current puzzle. Adopting this change reduced
the average solution distance from 3.68 to 3.51 guesses.
My partner is a research scientist working in a lab group. Every year, members
of the lab are expected to go on long driving trips to collect specimens for
research. This labor is shared among the members of the lab (in practice, the
graduate student members of the lab). So even though one lab member’s work
might only focus on a single species, a research trip might collect many
different species, even if the lab member who requires them is not present on
In practice, the planned destinations are known at the beginning of the
academic quarter. The trips need to be planned out in such a way that effort
and route duplication is minimized, while each individual only goes on as many
trips as their workload will bear.
While not a professional programmer or computer scientist by any means, I’ve
been tinkering with software long enough that coming up with ways to solve
problems like this is not that hard for me, and so I’ve been asked on several
occasions now to solve the routing problem for them. Having done that, I
thought I’d document my process (using entirely open source software and data)
and provide a few hacked together scripts to simplify the process.
The core piece of the puzzle is Vroom, a software
project explicitly designed to solve the vehicle routing problem in an
efficient way. It’s designed to integrate with
OSRM, a routing engine built with OpenStreetMap
data. If you’re curious about the system, I recommend this underappreciated
talk about Vroom.
Dealing with constraints can be complicated. Fortunately, I never had to worry
about blackout dates, because the routes once calculated could begin whenever
the driver was ready to start. That said, Vroom has one significant limitation:
the only metric it allows you to optimize is total drive time. If there is 3
weeks worth of work to do, and it minimizes total drive time to send one
vehicle on a very long trip, Vroom will do
You can’t tell Vroom to, for example, minimize the length of the longest
trip, or to make all trips as close together in length as possible. I thought
at first that this would make Vroom unusable for my tasks (it certainly doesn’t
seem like it would work for hot pizza delivery, for example), but with a lot of
handheld tinkering you can get it to work using availability constraints on the
vehicles. I also didn’t find any software that would handle the problem better.
While OSRM does provide a very generous free API, it’s still too limited for
the kind of routing I needed to do. Vroom works by requesting a routing matrix
from OSRM that contains the travel times between every pair of points you
submit. Sometimes I would be given a thousand or more destinations. This meant
I had to run a routing server myself. Fortunately I have an always-on server
that was more than up to the task.
You can find the routing server at
osrm-backend. There’s a Docker
image, but as I prefer to run software in different containers I simply built
it myself. The map data I needed (the US West subregion) is available to
download from Geofabrik.
After pre-processing, the data for the western United States is “only” 14 GB.
Very clear instructions for setting up the software are available on the
Github. You’ll probably want to use the MLD pipeline as the pre-processing step
is much faster. The page recommends the CH pipeline specifically for large
distance matrices (exactly what we have), but as we’re only going to submit a
small number of requests to this server in its lifetime, we can afford for
them to be slightly slower.
Vroom has a documented JSON format that describes the available vehicles, the
destinations they need to reach, and other constraints. I wrote a script to
create the necessary input for Vroom based on data in a spreadsheet (exported
to a TSV file).
The format of this file is
Latitude <tab> Longitude <tab> Skills <tab> Time
Skills is probably the most interesting feature I made available. Sometimes one
scientist is particular good at identifying one species, and so the plan is to
make sure they’re in the vehicle for every destination that involves collecting
that species. In the spreadsheet, the scientists can put a number (or several
numbers separated by a comma) in the corresponding Skills cell. Each vehicle
has a corresponding list of numbers which identify the scientists in each
vehicle (a scientist can go on multiple trips and so be in multiple vehicles).
Vroom will use the skills feature to only send a vehicle to a destination if
there is a scientist in the vehicle who can identify the specimen there.
The Time field is occasionally useful too. It’s an estimation of how long the
vehicle will spend at the destination not moving (in seconds). In most cases,
the scientists I was routing for would simply park the car, clip a branch of a
tree or shrub, and get back in the car, so this was effectively zero. In some
cases, however, the GPS point could be a mile or more from the road, meaning
that a longer hike was in order.
The scripts that I’m making available are very much hacked together, not up to
my usual standard for code I make public. For example, there are important
constants in the makejson.py file that have to be set or adjusted when the
script is run. I didn’t come up with a convenient way to use argparse for
We have the already-discussed list of VEHICLES:
VEHICLES = [[1,2],[3,4]]
This identifies two vehicles / trips, where two scientists (numbered 1 and 2)
are in the first, while two other scientists (numbered 3 and 4) are in the
We have lists of starting coordinates and ending coordinates for each vehicle.
These are in [longitude, latitude] format because that’s what Vroom itself
Note how the starting and ending coordinates for the vehicles are all the same
in this case. That’s a common pattern (starting and ending at the university),
but you’re not locked into doing that by any means.
Last we have TIMEAVAIL:
TIMEAVAIL = [400000, 400000] # seconds
This is the messiest part. It’s the total drive time allowed for each car. In
theory, this would be perfectly known to the participants in advance (they
could allow more time for stops, breaks, sleeping, etc., as needed). In
practice, though, it’s a balancing act. The research trips have to be done,
so even if finishing all of them would take more time than the scientists
theoretically have, put together, that just means that they have to find more
time from somewhere. So in practice the only workable approach that I found was
setting this value unreasonably high for each car, then reducing it
selectively to get routes that looked reasonable. When you can’t reduce it
any more without the router failing, you have a roughly optimal solution.
Vroom will print a large JSON file with the results (and the routes themselves
from OSRM in polyline format), and so I have another script that takes the
result and prints the result as a long list of coordinates for each vehicle,
and also exports the routes (as lists of coordinates) to text files, so that
they can be shown on a map. Theoretically I ought to create KMLs or something
more useful, but this turned out to be more complicated than I had hoped to do
in Python and so I ran out of time.
My repository contains a run.sh file that shows how I run the program for my
specific setup. Obviously you’ll need all the pieces in place (Vroom, OSRM on
a server, etc) in place for this to work. Running the script, even for a large
list of destinations, takes only 3-5 seconds, so re-running it while changing
the parameters is no big deal.
This solution shows three different routes, constrained to a limit of about a
week per trip. Several of the points have additional constraints, such that
only one scientist can collect the sample, and this has still allowed for a
nice solution. In many cases, it’s possible for human intuition to come up with
a map that looks a lot like this one, but getting the exact order of
coordinates right turns out to be crucial in saving hours of drive time.
At the end of the day, I suspect my exact approach won’t be completely workable
for most people. You’ll have slightly different constraints than me, a
different input format for your data, or some other problem. That’s why I’m
just sticking these scripts online instead of spending the time to make sure
they’re as functional and well-written as they could be. Hopefully, others will
be able to look to them for help in solving their own routing problems.