Adam Fontenot

Hi! I’m a graduate student in philosophy. Thank you for visiting my website, which hosts my personal blog. If you would like professional information or to contact me directly, please visit my about page.


LibreOffice 7.4 has a new approach to text rendering

18 August 2022


I’ve used LibreOffice for quick word processing tasks for years now — since before it was forked from OpenOffice in fact. However, for years I’ve drafted documents in another tool, before copying them into LibreOffice for the final export to PDF. The reason is that the font rendering in LibreOffice is infamous for how terrible it is, with a large number of issues reporting going back years.

As an example, one of these reports was mine.

Here’s a brief explanation of the problem. Fonts set the amount of space (called “tracking”) around each glyph. Certain combinations of characters do not look right with the default amount of space, and a feature called pair kerning allows a text renderer (like LibreOffice) to adjust this space for individual letter pairs. For instance, the large open bowl of the uppercase “C” looks too empty by default, so the next letter is often brought closer to it.

Unfortunately, LibreOffice’s rendering on low resolution screens was so bad that enabling pair kerning (it has been the default for many years) made it worse. In some cases the failure is catastrophic:

LibreOffice 7.3 rendering "California" with pair kerning

This is a real screenshot taken on my computer. I wrote software to upscale the image so that the problem is clear. Each individual (red, green, or blue) sub-pixel as shown on my screen is being re-rendered at nine times its original size, in grayscale. This helps to capture the effect of sub-pixel anti-aliasing, which uses the shape of the pixels on modern LCD screens to triple the effective horizontal resolution of the text. Here is the original screenshot, for comparison:

California, in its original zoom

Pair kerning is implicated here because it is actually this feature, when enabled, that turns “California” into “Califomia”. The font shown here is Liberation Serif 12pt, at 120% page zoom. Hinting is disabled system-wide.

Why is text rendering on LibreOffice 7.3 so bad?

The problem was caused by an approach to text rendering that predates the introduction of the codebase into version control in the year 2000. Back then, text renderers were rather stupid, and did not do a good job of rendering strings of text in a readable way on the screen. LibreOffice (then OpenOffice) therefore did its own glyph positioning.

LibreOffice knew where the glyphs should go (called the printing position, because it was where the glyphs would be placed when sent to a high resolution output like a printer), but just sending these “ideal” locations to the text renderer would result in subpar text.

On the other hand, if LibreOffice rendered the text at the preferred screen position, the width of the text would not match up with the printer’s output (because the screen metrics of the font differ, e.g. due to hinting). LibreOffice is a WYSIWYG (what you see is what you get) editor, so this is unacceptable. Lines that exceed the word processor’s margins must reflow, so you would end up with cases where the printed document would have different lines than the document as shown on the screen.

Instead the authors adopted a compromise: 75% of each glyph’s placement is determined by the ideal position according to the screen metric, and 25% is determined by where the glyph would be placed if its right border was aligned with the right border of the ideal printing position. WYSIWYG is made possible by realigning the text with the printing position every time white space appears in the text.

Unfortunately, the end result is not very good by the standards of a decade later. On a low resolution screen like mine, pixels are quite large relative to the size of the glyphs:

illustration of the size of one pixel

The above shows close to ideal kerning for the r/n letter combination for this font at 12pt on my screen. As you can see, being off by one pixel is enough to make “rn” into “m”.

Despite LibreOffice making its best effort to place glyphs intelligently, most of that work was thrown away at the last stage in the rendering process, because the glyph positioning engine did not support sub-pixel positioning. This is a different concept than sub-pixel anti-aliasing. While the font renderers used by LibreOffice have generally supported the latter, sub-pixel positioning support is much more recent. Sub-pixel anti-aliasing (sometimes more generally called sub-pixel rendering) is the use of the thin red, green, or blue components of LCD screen pixels to improve the resolution of text. Sub-pixel positioning uses anti-aliasing (potentially including sub-pixel anti-aliasing) in order to render text at positions more precise than a single pixel.

This is crucial to the proper appearance of text. Because pixels on low-resolution screens are so (relatively) large, rounding the location of glyphs to the nearest pixel generates consistently bad results. Combine that with pair kerning (which tends to move glyphs closer together, and also changes the heuristic described above to favor the printing position), and you’re asking for catastrophic failures.

What does LibreOffice 7.4 do differently?

For at least the last 22 years, LibreOffice has done its own glyph positioning under the assumption that the underlying font rasterization engines were not capable of creating high quality text on LCD screens when passed the “true” positions of each character (i.e. the printing positions).

This was true for most of LibreOffice’s history. After all, while rendering has been acceptable on Linux for at least a decade (with the proper settings and a recent enough FreeType), commonly used libraries like Pango / Cairo did not support sub-pixel positioning until very recently. Qt was ahead of the game for a long time, supporting it since some time around 2015.

During the LibreOffice 7.4 development cycle, Caolán McNamara experimented with this behavior, initially adding an option to just kick the printing positions down to the rasterizer and let it place them on the screen using its own algorithms. The result was successful enough and fixed so many bugs that it was made the default mode in LibreOffice 7.4.

Because the underlying rasterizers now support sub-pixel positioning, this change results in LibreOffice getting that for free, in addition to the intended change of eliminating the buggy positioning code from the render path.

The results are breathtaking:

gif of improvements with McNamara patches

It’s rare to see such an enormous improvement in a piece of software in a minor version bump, and this one appears to have gone almost unnoticed. It’s even rarer to see an improvement like this with a simplifying change, in this case one that makes a bunch of old code irrelevant.

My thanks to Caolán McNamara for working on this, and for backporting the patches to LibreOffice 7.3 at my request. LibreOffice is now a viable choice for me when drafting documents.

I hesitate to share the following before and after picture of some text at the original size for fear it may be misconstrued. Don’t expect this text to look great on your system, even if you view it at 100% zoom (which you should — right click to open the image in a new tab). Font rendering happens at a very low level, relying on details like screen DPI and pixel orientation that may vary from screen to screen. This is also a bit of a stress test, rendering some serif fonts with many details on a low resolution device at 12pt.

comparison between 7.3 and 7.4

Rather than deciding which rendering looks better to you overall, focus on the differences in glyph placement. Where are the glyphs too close together? Where are there ugly gaps? To my eye, LibreOffice 7.4 has significantly reduced severe positioning errors (“met” in the fourth line of the Garamond sample) while achieving superior rendering even in cases where LibreOffice was already acceptable (every “th” pair is slightly too close in Caslon), thanks to sub-pixel positioning.

Statistics-Assisted Weight Loss with PyWeight

11 April 2022


Content note: this article contains a discussion of weight loss and healthy eating, and includes my before and after pictures at the end. You can view a version of this document without these pictures here.

Quick summary

I lost 80 pounds following a diet focused on calorie restriction and satiety by writing software to track my weight and suggest adjustments. The program is PyWeight and it is available as a free download for Windows or Linux.1

The premise of the diet is that it’s possible to achieve a precise rate of weight loss when restricting your intake without counting the calories in all the food you eat or knowing your daily energy expenditure. A reasonable estimate of how much you’re eating is good enough, when assisted by technology.

Because a calorie deficit and the resulting rate of weight loss are mathematically related, software that measures your average rate of weight loss can accurately determine whether you’re eating too many or too few calories, even if you don’t count them exactly. You can use the information the software provides to hone in on your desired rate of weight loss by making small adjustments to your diet over time.

If this sounds interesting to you, read on.

People who want to follow this diet should talk to a doctor beforehand.

Why everyone is gaining weight and why it’s so hard to lose it

Some people believe that their bodies are genetically programmed to reach and maintain weight at a particular “set point”.

This clearly can’t be the whole story. In the last 40 years, the average weight of the adult population in most western countries has increased significantly. If these weights reflect our natural set points, then we are forced to conclude that just about everyone else who ever lived somehow never reached their body’s natural weight. We’re the lucky few — except that apparently we’re unlucky, since this supposed biological imperative just so happens to be bad for us!

Yet neither can the set point view be completely wrong. It accords too well with several common experiences.

When someone in a country without a fast-food culture moves to one with an urban environment, they frequently gain a moderate amount of weight and then stabilize at a new point from which they find it difficult to gain or lose.

Similarly, people who do manage to lose weight often hit plateaus — low points at which they find further weight loss to be much more difficult. Some people regain lost weight until they’re about where they started. What’s really happening?

I suggest that the truth lies somewhere in between. You do have a distinct set of variables that dictate the weight you will approach in the long term. These include your genetic makeup (with its effects on your metabolism, skeletal structure, and so on), age, and hormone levels. But these variables also include diet and exercise, two variables that are partially under your control.

In the rest of this article, I will refer to the weight you are approaching given your current daily intake (and other variables) as your “equilibrium weight”, to distinguish this from the simplistic “set point” view.

Holding everything else constant, if you are currently maintaining your body weight and you permanently reduce your intake by 100 calories per day, you will eventually stabilize at a new equilibrium point about 10 pounds lighter than your current weight.2 A cut of this size is equivalent to drinking one fewer can of Coca-Cola each day at work (five days a week).

Why losing weight is hard, and maintaining weight loss is even harder.

There’s one additional but very important wrinkle: one of the variables that dictates your equilibrium weight is your personal history of weight changes, because of its effects on your metabolism and body composition.

Consider three genetically identical people of my height and age. They all weigh 235 pounds.

According to one model of diet performance3, they will need to eat as follows to maintain their present weight:

In other words, if all three want to lose weight, the person with the thin “history” has an advantage. Because their body has not yet adjusted to carrying this extra weight, they will burn an extra 400 calories a day for free.

A 400 calorie per day difference is a whopping huge one. It’s about the equivalent of eating a pint of your favorite Ben and Jerry’s ice cream every Monday, Wednesday, and Friday. On top of that, the previously 350 pound individual was able to eat about 3800 calories a day at maintenance. So just to maintain their current level of weight loss, they have to get used to eating 800 fewer calories than they are used to.

This may seem unfair. These are people who are genetically identical, the same age, the same height, and the same weight. But one of them will maintain at 400 calories a day higher than the other. It is a little unfair, but remember that these are ultimately temporary metabolic effects. Eventually Person 3, who gained 85 pounds, will become Person 1, who has carried that weight for a long time.4 The body doesn’t provide you with a metabolic advantage forever.

This leads to two important principles for maintaining weight loss:

Most people ignore one or both of these principles. It’s tempting to treat diets as one-time interventions similar to a medical procedure. This simply will not work, as the foregoing discussion indicates. You can’t eat like an overweight person who has temporarily starved themselves to fit into a smaller pair of jeans. What you must do if you want to keep weight off is to permanently eat like the thin person you now are. You must lower your equilibrium weight through consistent, long term improvements in diet and physical activity.

Most people recognize on some level that when they regain weight after dieting, it’s because they “ate too much again”. What many do not realize, however, is that “too much” means something different after dieting than it did before.

The worst part about all this is that you’ll begin to feel the effects of the two principles while you’re still trying to lose weight.

Most people start a diet expecting to lose weight at a relatively quick rate — and they do their first week or two. Unfortunately, most of this weight loss is fake. Running a calorie deficit means that your body burns through its store of glycogen, which is what causes it to tap your energy reserves (fat). Along with the loss of this weight, you will see a one-time drop that’s just water weight; your body has less food temporarily stored in your stomach and intestines as it’s being digested. Your body will also adapt to reduced intake relatively quickly, and your metabolic rate will drop slightly.

Metabolism is not magic. Your body obeys the law of conservation of energy when your metabolism “slows”, and it doesn’t have a more efficient “starvation mode” that uses fewer calories to perform the same tasks. (If it did, natural selection would surely have dictated that we would always be in that mode, and thereby need to eat less.) Rather, a “slowed” metabolism means that you will either be slightly colder (unlikely to be by enough to make a serious difference) or that you will have less energy available, and therefore perform fewer tasks. Maybe you will feel like exercising less, or maybe you will perform tasks in a lazier fashion, or maybe you’ll simply sleep more. Putting all this together, it means that you can expect to lose some fraction of your caloric deficit to temporary metabolic changes.

If you don’t get discouraged and continue dieting, your weight loss will continue apace, but unfortunately you start to run into the problem discussed above: your reduced size is resulting in a smaller energy expenditure. Research suggests that for most reasonably large deficits, it would take years for this effect to neutralize further weight loss (at which point you would have achieved a new equilibrium), but in the mean time your losses each week will get smaller and smaller. No wonder most people get discouraged! They end up cheating or adding calories to their diets little by little, and eventually they stall out. This is why many people hit plateaus in their diets. A slowing rate of weight loss leads to discouragement, and discouragement leads to a failure to adhere strictly to the diet, and this leads to you hitting a new equilibrium point much earlier than expected.

Because of your body’s remarkable capacity to adapt to changes in food availability, you have only two choices if you want to overcome this problem. You can start out with a lower calorie budget, precisely calibrated to match the metabolic changes you anticipate. Or you can adjust your calorie intake as you go along, to maintain a constant deficit. I strongly suspect that the latter approach is easier to follow and less discouraging, so it is the approach PyWeight is based on.

Where is the obesity epidemic coming from?

This talk of equilibria makes an obesity epidemic rather surprising. Even if people are eating more than before, why aren’t they simply reaching new slightly higher equilibrium points, like our theoretical immigrant? After all, if the way one developed a body that required 4000 calories just to maintain itself was to start eating 4000 calories a day, no one would ever do that. Most people with normal body weights couldn’t even stomach 4000 calories for many consecutive days.

But the evidence for epidemic obesity is clear. Here in the U.S., the state of Mississippi had the highest rate of obesity in 1995, at 19.4%. In 2010, a mere 15 years later, Colorado had the lowest rate of obesity in the country, at 19.8%, and 12 states were over 30%.5 Today, more than 40% of American adults have obesity.6 This is happening quickly enough that it doesn’t work to posit genetic or behavioral changes in a new generation of adults: the same people who were thin 10-20 years ago have obesity now. So what’s happening?

Of course, when we’re talking about the obesity problem, we’re talking about averages. Some people do maintain an equilibrium point that’s just overweight. Other people don’t become overweight at all. Yet in this country, we seem to have reached the point where the majority of people put on weight year over year, endlessly.

You can square an equilibrium view of body weight with this fact using the suggestion that we’ve reached some kind of tipping point. For some reason, people are driven to eat slightly more than they need to maintain, and for many of them the rate at which they increase their intake has begun to outpace the body’s compensatory mechanisms.

The body is calibrated to give you hunger signals that will result in you eating approximately what you need to eat to maintain your current weight, on the sort of diet most people had in the past. Someone who could maintain at 2000 calories a day and significantly overate7 would gain weight, but after doing that their body would still only be giving them hunger signals appropriate to maintain their new body weight — or more precisely slightly less until they adapted. (Effectively, the two principles mentioned above are functioning in reverse here.) So this person would not automatically continue to gain weight. The body’s negative feedback of constantly feeling too full would tend to keep their weight in a healthy range.

Our hunger signals are no longer accurate. The body still says “I need 2000 calories”, but it doesn’t feel full until you’ve eaten 2100, thanks to the low satiety calorie-dense diets of today. When you gain enough weight on such a diet, your body will start to say “I need 2100 calories”, but now it takes 2200 to fill you. And because eating food takes time and costs money, your increased appetite will be drawn towards quicker and cheaper options, further exacerbating the problem. Before, the body gave you you a signal that you should eat less food the day after a big meal. Now, even a big meal isn’t enough to make you feel full.

This suggests we should change our view of personal responsibility. People in the last 30 years are facing a problem that virtually no one in the past faced: their stomachs are lying to them about how much they should eat.

Recall the equilibrium equation: your equilibrium weight is a function of your genetics, body composition, age, hormones, diet, and exercise. If you consider this equation on a societal level, you can replace diet and exercise with just one variable: environment. In the absence of a strong effort to control these two variables, the health environment you are in will dictate them to you, and therefore control your weight.

Faced with the environment that most people in Western countries faces, the average person is going to gain weight over time. So on a broad social level, that’s what we see. Blaming individual people for the consequences is not just counterproductive, it’s an inaccurate accounting of what has happened.

Consider the following scientific model of average weight gain8:

What this model indicates is that the weight gain of the average American over the last decades can be explained by a relatively tiny gap between energy intake and energy expenditure. In fact, the average gap for the period examined in this study was only about 7 calories a day. The problem doesn’t amount to 7 calories here or there, the epidemic stems from the fact that this gap is so reliably sustained. High calorie junk food is causing many of us to overeat, day after day, with no feedback telling us to cut back for a while. This has gone on for so long that the average person (as of 2005) needed to cut out more than 200 calories a day to achieve the equilibrium point of the average person in the 1970s. This number, of course, has only continued to rise.

Watch this excellent five minute clip detailing some of the causes of calorie overproduction in the United States:

Source: The Weight of the Nation, 2012. HBO Documentary Films.

You might be tempted to decide the takeaway of this analysis is political: corporations are profiting off of human suffering on a massive scale, so some social action is needed to fix the problem. Insert regulation, social democracy, or whatever your preferred solution happens to be. That’s all well and good, but it’s of little use to someone suffering the effects here and now. This country will happily let you die in the street. In all probability, things are not going to get better on a timescale that matters to you. The only option is to take matters into your own hands and exploit the two variables under your control to the maximum extent you can. For some people, that will be enough.

The good news

The one bright spot is that for most people the food environment is not yet bad enough to make it impossible for them to control their weights. If you are not very constrained in your diet and exercise choices, then you have a fairly good chance of overcoming your environment.

As for the other variables in the equilibrium equation that are outside of your control (e.g. genetics and metabolic effects), the good news is that these almost always have a relatively minor impact on your weight. Most people have a basal metabolic rate (the number of calories you burn each day just by being alive) within 10 percent of the rate estimated by the Mifflin-St Jeor equation. “Slow” metabolism is not usually the cause of weight gain.9

This means that many people can change their equilibrium point by managing their calorie intake and becoming more physically active.

Addendum: it’s okay to try to lose weight, I think

Some authors, including philosopher Kate Manne, claim that encouraging dieting and even dieting yourself are immoral. I agree that negative self-image and exploitation in the diet industry are both things we should condemn, but for this to have consequences for individual dieting decisions something more is needed.

I think that’s why Manne’s article, though it defends an ethical conclusion, ultimately rests on two empirical claims. She writes that “fat bodies can be not only healthy but also athletic” and “the vast majority of diets fail to make people any thinner or any healthier in the long term”. These claims are both true, and suggestive, but don’t on their own deliver her conclusion.

It is true that some fat bodies are healthy. Similarly, many people who don’t get the recommended 30 minutes of exercise every day are also healthy. Despite this, since exercise and a reduction in overweight tend to result in health improvements, it can make sense to pursue both.

Likewise, the fact that most diets fail cannot deliver the conclusion either. If this claim is to show that it would be a bad idea for me to start a diet, it needs to be true that I can have no expectation of success, so that trying to lose weight is similar to playing the lottery. I think this is false. There are reasons that most diets fail, but dieting is not tantamount to fighting genetic or environmental determinism, as I argued in the previous section. I explain further why I think it is reasonable to expect success in the next section.

I suspect that any versions of Manne’s empirical claims strong enough for her conclusion will be false.

The claim that some fat bodies are healthy suggests a stronger version that would suffice: most fat bodies are healthier than the same bodies would be if they were thin. Dieting would then, in all likelihood, make you less healthy. In this form, the route to Manne’s conclusion that encouraging dieting or dieting yourself “is a morally bad practice” is obvious, but the premise seems clearly false. There certainly is a correlation between body weight and health. As I will discuss later, some (contentious) studies suggest that being slightly overweight does not increase mortality, but no medical authorities will endorse the notion that obesity is healthy.

Likewise, if diets could be shown to be inherently dangerous, unhealthy, or doomed to fail, it would be easy to reach the conclusion. Manne hints at this: she says that in one month early in her diet, she didn’t eat for 17 out of 30 days. This is indisputably a bad approach to dieting. If most people trying to lose weight go for diet fads and ineffective products, or take dangerous or disordered approaches like extreme fasting, that would help explain the failure rate.

However, it seems false that these problems are inherent to dieting. I ate food every day on my diet. Manne herself seems to have ended up with a sensible diet: she lost about 50 pounds in a year, close to the common recommendation of a pound per week. The diet industry is certainly blameworthy for distracting people from methods likely to be successful and promoting fads, but this alone can’t show that dieting is wrong.

How to succeed at dieting

Most diets fail. In a major longitudinal UK study, less than 20% of people with obesity managed to decrease their BMI by at least one category without a subsequent increase, over a 9 year period. 78% of people who lost at least 5% of their body weight gained enough back that after 5 years they were back within 5% of their original weight, or even higher.10 This indicates that while plenty of diets do succeed (the popular claim that 95% of diets fail is incorrect), successful dieters are by no means the majority.

I don’t believe a 20% success rate should be discounted. It suggests that diet success is possible and that we should be looking at what the successful people do. If we were to discover that a particular intervention to help people quit smoking had a 80% recidivism rate, we would still encourage people to take the program (because 20% is much better than zero).

Part of the issue with assessing the success of a typical diet is that if you lower the bar of what counts as successful weight loss, then the success rate goes up. If you demand studies involving weight loss of a quantity closer to what I was hoping to lose, examples become even rarer. Most studies are testing specific short-term interventions to see what works and what doesn’t, not looking at how to achieve a long-lasting high level of weight loss. It’s hard to get reliable data about the prevalence of successful diets resulting in long term weight loss of 25% of body weight or more. A large portion of attempts likely fail.

The reasons why most diets fail go beyond simple misunderstandings of what it takes to succeed, of course. Food is tempting. Not eating food when you are hungry sucks. The restrictions that a diet imposes can be difficult to implement. The nutritional deficits imposed by Western eating can become even more severe on a calorie restriction, especially for those who can’t afford to make their own food. For many people, a personal trainer and gym membership are unaffordable, and for some even a doctor visit is out of reach. (It was for me when I began my diet! Thanks, America.)

Given the high rate of failure, what I went looking for was a diet that could guarantee success to those who stick to it and and don’t cheat. My hope was that following such a diet would not only bring me the weight loss I was looking for, but that the certainty of success might provide additional motivation to stick with the diet.

As a result of studying physics in college, I was naturally inclined to follow a “calories in, calories out” diet. This comes with something like a success guarantee: if you consume fewer calories than your body uses each day, you will lose weight. Other diets may be successful as well, but only if they end up resulting in calorie reduction anyway. Diets may have some individualized metabolic effects but these appear to be minor in comparison.1112 Perhaps some other diet is easier to maintain than calorie restriction, but no other diet could provide the guarantee of success.13

Counting calories has a reputation for being difficult, which is part of the reason diets that dress up the calorie counting by using “points” systems are so popular. It’s not only easy to underestimate the number of calories that you’re eating, it’s also difficult to determine with accuracy how many calories you’re burning. Moreover, if you weigh yourself once a week or so, your measurements might get swamped by variations in water weight or having an unusually large lunch, so it’s hard to know if you’re making progress. If you weigh yourself once a day, the value on the scale is likely to fluctuate so much that you constantly fly into a panic over an imaginary weight gain.

Furthermore, you might already be disheartened by reading the previous discussion of why it’s hard to lose weight. As you lose weight, the number of calories you need just to stay at maintenance is going to be decreasing. How is it possible to reliably stay ahead of the curve? It turns out that there’s a simple trick that can be summed up in a single paragraph, and this trick obviates most of the complexity of weight loss. This trick is:

Record your weight every day. Every two weeks, calculate your approximate weight loss using your record. If you are losing weight too quickly, eat more the next two weeks. If you are losing weight too slowly, eat less the next two weeks.

That’s it. Weight loss, summarized. By only worrying about whether your rate of weight loss matches your predetermined goal, and not calories or points or what have you, you can hone in on a reliable rate of weight loss over time with relatively little effort. Accurately counting calories or determining your daily expenditure is not required.

You can do this yourself with just pencil and paper if you like. But technology will help you in two ways: you can more accurately calculate your rate of weight loss using techniques like linear regression, and you can estimate how many calories you missed your target by (instead of pounds), to make it easier to adjust your intake in the future.

This idea was inspired by the book The Hacker’s Diet. The book’s version of this scheme goes something like this: if we assume that each pound of fat lost contains about 3500 calories, then the math becomes quite simple. If you need to lose x pounds, and you maintain a deficit of about y calories a day, it will take you about 3500 * x / y days to do that. At a deficit of 500 calories a day, you will lose about a pound a week. If after two weeks you’ve only lost 1 pound, you know you’re missing your target by 250 calories per day.

Unfortunately, this relies on the myth that a pound of fat contains 3500 calories. It turns out the correct number is closer to 4280.14 The slightly better version of the theory, which says that the average caloric density of the weight you lose is 3500 calories, is an inaccurate approximation. You do lose some fat free mass while dieting, and this contains around 824 calories per pound. So 3500 calories per pound is only accurate if about 3/4 of the pounds you lose are from fat. This is only a reasonable estimate for very overweight people at the beginning of a diet. Later on, as you approach an optimal body weight, you will lose a larger proportion of lean mass. If you were to continue losing weight indefinitely, you would of course eventually have nothing to lose but lean mass. This would be extremely bad for your health.

Because of this complication, PyWeight uses an equation from a real diet model15 to estimate how many calories were contained in the weight you lost in a given interval, and therefore how many calories you are away from hitting your weight loss rate target. This means it’s possible to target a linear rate of weight loss despite all the non-linear effects. (For what it’s worth, when you’re targeting a constant rate of weight loss rather than a constant, pre-determined calorie budget, the problems caused by inaccurate estimates are greatly lessened. In fact, you’d probably be fine with the old 3500 calories per pound estimate. The upshot of PyWeight’s advice is usually just “try to eat a little less / more next week”, just as it would be if you were using pencil and paper.)

With the Hacker’s Diet, the calculations are all done in a series of complicated spreadsheets which are provided with the book. PyWeight allows entering your measurements directly in a built-in log, and all calculations are done for you automatically. It also provides intake adjustment advice at a user-selectable frequency.

The linear regression method used by the Hacker’s Diet can also be improved upon. Doing a naive linear regression by breaking the data into intervals lasting 2-3 weeks means that only the data from those weeks figures into the calculation of your current rate of weight loss. But this is throwing away data; you can determine a much more accurate starting weight for each interval if you also make use of the data from the previous interval. PyWeight takes exactly this approach with a technique called spline interpolation. A possible alternative would be to use a moving average, but I believe this would give worse results because of the high degree of variability in weight measurements and the fact that the diet is built around attempting to lose weight at a constant rate.

As a side effect of this improvement, PyWeight is able to generate very nice looking graphs of your weight loss. Here is mine from the last ~7 months. (I lost the first 20 pounds using the methods described in this article before I wrote PyWeight.)

my graph in the pyweight program

The idea behind using a program to track your weight is that despite the fact that our hunger systems are broken because of modern calorie dense diets, we can use technology to replace this body function, letting simple math regulate how much we eat. This method actually works even better than simple calorie counting, because it adapts to your changing energy needs as you lose weight. If you want to keep up a one pound per week deficit, you’ll need to reduce your intake over time, and this approach will help you do so automatically.

Even though you’re measuring your weight every day, you don’t have to worry about what the scale says in the way you do with ordinary calorie counting. An unusually large meal might send your apparent weight up 1-3 pounds for several days, but this is mostly water weight that will soon go away. Looking at a statistical measure of your weight loss over a period of weeks smooths out these misleading bumps in the graph.

Looking at my graph, you can see several of these blips, some of them quite large. My daily scale measurement increased 6.8 pounds over the course of four days in late December, but this did not represent any long term weight gain.

I’m not going to tell anyone that this approach makes dieting a sure thing. A great deal of willpower is still involved. But knowing exactly what you’re doing and trying to keep up a consistent rate of weight loss makes it much easier. Steven Novella helpfully writes:

I would conclude that weight control is difficult, but not impossible. In addition, I believe that the popular weight loss industry is making the problem worse by distracting the public from those strategies for which there is at least some evidence of efficacy and focusing on minute, short term, and probably insignificant effects from manipulating macronutrient proportions.

My goal with this diet is to focus on what we know works (consistently eating fewer calories than you burn) and find a way to make sure I am doing that with small course corrections over time. I began this diet with a high degree of confidence that it would work if I didn’t get burned out and give up, and that confidence was rewarded. As you can see from the graph above, I was able to maintain an extremely consistent rate of weight loss over time; I never deviated from my monthly target by more than a few percent (except around the holidays and a strenuous four day hiking trip in mid-February during which I snacked almost continuously).

It’s not just losing the weight that gives me confidence this method works. It’s the fact that I lost the weight at almost exactly the rate I planned in advance! I think many individuals can see success in dieting if they can follow a similar approach.16

No method for individual correction can solve the underlying problem. So long as the United States fails to get a handle on its poor nutrition environment, obesity will continue to be a serious issue. Proposing diets as the solution is like thinking that encouraging people to recycle can prevent climate change. However, until the social causes of obesity are resolved, individuals are on their own. Diets will not reverse the long term obesity trend, but they can help dedicated individuals who are fortunate enough in their circumstances.

Setting goals

I started out just above 240 pounds, which at my height was a hair into the obese range (30+) by the BMI metric. BMI is not necessarily a good indicator of individual health, but the truth is I felt bad. I did not feel like exercising, or even going for a walk. Hiking long distances, one of the activities I’ve enjoyed the most over the years, was certainly out of the question. While it’s not impossible to be in good shape at the size I was, it’s certainly much more difficult.

How bad is it, in general, to be overweight? There has been some recent controversy on this point in the medical world. The long-time consensus has been that while individuals with BMIs between 25 and 30 are not necessarily unhealthy, overweight is correlated with heart disease and diabetes, as well as increased all-cause mortality.

One 2005 study suggested that being in the overweight category was associated with reduced all-cause mortality relative to the normal (18.5 - 25) BMI category.17 A follow up meta-analysis18 by the same primary author confirmed this finding, and also found the difference between mild obesity and normal body weight to be insignificant. However, a much larger third analysis19 disagreed, finding that after controlling for smoking, prior chronic disease, and length of follow-up, both overweight and obesity were correlated with increased all-cause mortality risk. Moreover, they were able to reproduce the findings of the second paper by removing those quality controls. (The danger of leaving in smokers is that a disproportionate number of smokers will have healthy body weights because nicotine functions as an appetite suppressant. Smoking is much worse for you than a few extra pounds, so these people make “normal” body weights look less healthy on average.)

I’m not here to say that the third study was right and the first and second were wrong. I don’t have the expertise to do that, to begin with. The primary author of the first two papers wrote a response paper which claimed that the approach taken by the third was biased and suggested that the exclusion of smokers might lead to more bias than leaving them in.20 The result has been an increasingly bitter academic feud over claims of “bias” with no obvious conclusion (to me).

However, a very nice fourth paper21, a systematic review of the issue, reached more or less the same conclusions as the third. It was significantly larger than either of the others, containing almost 10 million individuals in the never-smoking category alone, and none of the authors overlapped with those of the third study. This study found that an ideal BMI was approximately 23, with negligible differences between BMIs from 22 to 25. Furthermore, the paper reproduced the most important finding of the third: the conclusion was reversed when filtering was removed. A BMI of 27.5 was associated with a lower mortality risk than one of 23, and a BMI of 32.5 lower than one of 20!

Because this study subdivided “normal” BMI into many categories, the conclusions are a bit more fine-grained. The results may help explain why in some studies the 18.5-25 BMI range compares unfavorably with the 25-30 range. The dangers of being underweight are quite extreme: among never-smokers, the study found that a BMI of 17.5 was associated with a 35% increased mortality risk. Someone with a BMI of less than 20 contributes to the average outcome for the normal BMI range (if their BMI is 18.5 or higher), but they’re exposed to more of the risks of underweight compared to someone with a BMI of 23, which the study found to be optimal.

I believe it’s very important for casual interpreters of medical research to remember to look at papers in context. All this bickering is essentially over a 5-10% difference. If you believe the fourth paper, a BMI of 27.5 is associated with a 7% increased all-cause mortality risk versus a BMI of 23. If you believe the second paper, having a normal BMI might be associated with 6% more all-cause mortality risk versus an overweight BMI. Personally, I’m inclined to believe the former, because the studies are larger, it’s more in line with the medical consensus view, and because the exclusion of smokers in the analysis seems justified. But I see no reason for me to argue with someone who believes the latter. To my mind, the sane middle ground is found by keeping your BMI well out of the obese range, and at a level where you feel able to keep yourself in good shape. My guess is that the difference between the two is swamped by just getting plenty of exercise.

For me, the truth is that I wasn’t able to feel healthy or stay fit at the weight I was, and the BMI number merely served as a painful (though possibly useful) reminder of that.

For the sake of having a goal weight, I picked the arbitrary target of 160 pounds. This happens to correspond nicely to taking 10 points of my BMI, and struck me as unlikely to be unsafely low. It corresponds to a BMI of slightly above 20, and my plan is to put on a few pounds of muscle over the next year. The last time in my life when I was thin was around the age of 10 or 11 (when I had temporarily become a vegetarian), and I wanted to experience that again.

It remained only to choose a rate at which I was going to try losing the weight. The UK’s National Health Service says that a weight loss rate of 0.5 to 1 kg a week is safe. 1 kg a week is equivalent to 9.45 pounds lost every 30 days. I rounded this up to a target of 10 pounds per 30 days. I’m considerably taller than the average person, and so there is a larger gap between my daily energy expenditure and my basal metabolic rate than for the typical dieter. I think this was a safe target for me.

I emphasize that if you’re going to pursue fast weight loss, you should talk to a doctor about that. You may very well have more success and invite fewer risks by targeting a pound per week (for some people, even less). In my case, without access to professional advice, I made the best judgment I could about what would be safe and effective for me.

How to get enough nutrition on a diet that doesn’t suck

It can be helpful to get a rough estimation of how many calories you should budget for yourself. In my case, something like 1800 calories a day at the start of the diet was appropriate. One nice thing about taking a statistical approach to weight loss is that even if you screw up this estimate quite badly, you’re unlikely to put yourself in any danger as long as your target rate is reasonable. If you lose too much weight in the first weeks of the diet, you will receive a sharp corrective directing you to eat several hundred more calories every day. PyWeight is thus self-correcting.22

Cutting a thousand calories out of your usual intake can make it difficult to get the nutrients your body expects to have available. It’s important to have a plan for getting the necessary vitamins, minerals, and essential amino acids.

This is an area where The Hacker’s Diet is not very helpful. It says

Since we’re efficient food processing machines, it’s possible to reduce all the complexity of food to a single number that gives the total energy the body can extract from it—the calorie.

The extreme form of this is the infamous Twinkie diet. You will lose weight on such a diet; it might also kill you. (It’s interesting that the actual Twinkie diet guy didn’t just eat Twinkies. A full third of his calories came from fruits and vegetables along with a protein shake. Eating no protein at all for months on end would probably not have gone well.)

To be fair, the book also suggests eating a balanced diet. As a vegan, I think there’s some cause for me to be concerned, especially on a diet as restricted as mine. I took a vegan multivitamin (DEVA tiny) every day of the diet, to ensure I wasn’t vitamin deficient. I really like this multivitamin because (a) it provides ~100% RDA of vitamins I was likely to be deficient on, instead of 10000% like some multis with potential side effects, and (b) it was small and easy to swallow. I also have a B-12 supplement that I take about once a week (as do most strict vegans).

So the remaining concern was protein. Getting enough protein isn’t generally a problem for vegans23 and even on a restricted diet it wasn’t one for me either, at least in terms of quantity. The issue is that there are minimum amounts of specific amino acids that we have to consume for building cells and other important stuff our bodies do. I wanted to make sure I was hitting the WHO minimum recommendations fairly consistently, even though I didn’t think a mild deficiency for a few months was likely to be harmful. (After all, the Twinkie guy didn’t fall apart.)

In order to hit these targets, I ended up on a diet that was pretty protein loaded. I hit the recommended amino acid intakes almost every day, even though I was on a diet with considerably fewer calories. At the same time, I also emphasized eating a lot of fiber and low-density foods. The point of this is that both fiber and foods with a high bulk to calorie ratio will make you feel full. Note that neither of these strategies will directly cause you to lose more weight than otherwise. They’re simply ways to make a low calorie diet easier to maintain by reducing hunger.24

Many vegan foods are complete proteins (that is, they include all of the essential amino acids). Most aren’t perfectly balanced between the amino acids, so you might have to eat more than enough of one to get enough of all the others. (This is one of the reasons why a balanced diet is often suggested, since eating many different protein sources will help balance the ratios out.) Note that this problem isn’t unique to plant-based diets — beef isn’t perfectly balanced either, nor are most other animal-foods. It’s only something to worry about for someone on an intense calorie restriction.

In trying to come up with meal plans, I found WolframAlpha indispensable. You can enter an amount of food, and it will tell you how much of each amino acid it has (assuming it’s in their database).

Lentils and soy turned out to be useful for hitting my high protein and fiber requirements on a low calorie diet. It’s actually possible to hit the WHO amino acid recommendations and much more than 100% of your fiber using mostly these sources on a mere 1200 calories a day.

You don’t have to eat the same thing constantly on this diet. (In fact the whole point is that you don’t need to be precise about what you eat or count calories exactly.) However, if you’re cutting so much that you need to be concerned about nutritional deficiencies, planning your meals may help. It’s probably still best to mix it up some of the time, just to ward off any additional danger of malnutrition.

On many days, I ate whatever kind of food I wanted (or even went out to eat) and just went with a rough estimate of my calories. The point is that as long as you are hitting your nutritional requirements, and can get even a rough estimate of the number of calories you eat, you’ll be fine. The best part is that you don’t even have to write anything down if you don’t want. It’s a calorie “counting” diet, but all the counting is quietly being done automatically for you by enhancing the quality of your estimations over time until they’re about as good as a real count. (Of course, should you struggle making the necessary adjustments to hit your targets, you might find a food diary indispensable, at least for a short time.)

If it is at all feasible for you, I do suggest making as many of your own meals as possible, and making them as highly nutritious as you can. Fast food is far worse than you or I probably realize, and it tends to be extremely calorie dense without satisfying.

Keys to success

I think adopting the following attitudes and practices while dieting helped me achieve success:

  1. Commit to a lifestyle change, not a brief period of restriction.

    As discussed above, this is actually mandatory if you want to keep the weight off afterwards. This means that you need to recognize that you’ll be giving something up (you will have to permanently eat less food), and decide that the cost is worth it.

  2. Commit to weighing yourself every day.

    Forming any habit can be irritating, but when your body has been sending you dishonest signals for years, this is really the only choice. Many people who are successful at maintaining weight loss continue to weigh themselves regularly for years afterwards.25 You’ll need to do so at least until your hunger system is retrained.

  3. Don’t declare any food off limits (other than pre-existing dietary restrictions).

    There’s no good reason to set yourself up for a cycle of failure and self-blame. One of the most common reasons for dieting failure is that people prohibit certain foods entirely, crave them incessantly, and then binge when the pressure proves too much or they have a “cheat” day. The resulting feelings of shame may drive them away from the diet entirely, especially if the number on the scale goes up the next day. (Usually, this is a temporary result of eating a lot of food. The whole point of using a statistical measure of your weight loss rate is that you don’t have to worry about the daily fluctuations!) Throw out the concept of cheat days; every day is a diet day, but every day is also a day that you may have a doughnut if you absolutely must have one. As long as the line on the weight chart keeps going down at the rate it’s supposed to, you have nothing to worry about.

    Try to develop a more healthy decision making process about what you eat. Ask yourself: am I willing to trade being hungry later for eating this sugary thing now? Speaking for myself, I ate almost no added sugar during the diet. It was never prohibited, though. I put a teaspoon of real sweetener in my tea. I had a little square of a chocolate bar once every week or two. I had a slice of pie or a few cookies over Thanksgiving and Christmas. I had a beer every now and then. Nothing off limits.

  4. Never give up.

    There will be days when you want to. If you have a tendency towards anxiety or depression, or you use food to cope with problems, you are going to have to face that head on. You’ll have to handle feelings of hunger and reduced energy. It’s even likely that there will be days where you screw it up. The difference between the people who succeed and those who fail is that people who succeed don’t tell themselves that they failed. Failure isn’t what happens when you have one bad day, it’s what happens when you give up.

Conclusions

I lost more than a third of my original body weight. The guarantee that you will lose the weight if you follow the simple instructions output by the program and dutifully weigh yourself every day was enough to push me through the hunger pangs and inclination to give up. Here are the usual before and after pictures.

Picture of me from 2019
Picture taken August 2019 at ~243 pounds. I was in a wedding.
Picture of me from 2022
Picture taken March 2022 at ~160 pounds. Appearance by choice this time. ☺️
Picture of me from 2022
Picture taken March 2022. This is the belt I was wearing in the first picture.

Is this a good way to lose weight? For me, the dominant consideration in answering this question is “is this a successful way to lose weight?”. That’s because most diets are unsuccessful, and so successfully losing a significant amount of weight is already a high bar to clear.

I think it’s reasonable to say that this can be a safe way to lose weight. If losing the standard recommendation of 1-2 pounds a week is a safe rate of weight loss for you, PyWeight will not tell you to do anything stupid. It is merely a tool to help keep you on track. You can choose a slower rate of weight loss and it will come off just as reliably, but will require more patience.

The other question, of course, is can I keep it off? That remains to be seen, but I’m planning to continue weight tracking for the rest of my life while eating at maintenance (net-zero calories in-out every month). This is something PyWeight can also help you do. If you set your desired deficit to zero, any time you begin to gain weight PyWeight will quickly warn you to reduce your intake.

The fact of the matter is that even if I knew for certain that I’ll gain it all back within five years, I’d still do it all again. I really do feel that much better, just as a result of the weight loss. You can’t assess the quality of someone’s life by what they happen to weigh when they die; even if I eventually regain weight I will have bought myself years of enjoyment in the mean time that can’t be taken away. That said, I understand that most people are looking for long term weight loss. I hereby commit to publish an update to this article with my current weight one year from the date this was posted, and then again a year after that.

Thanks to anyone who took the time to read all the way through this. Feel free to ask me any questions that you have.


  1. PyWeight is free and licensed under the GPL v3, which means you can modify it under certain conditions to meet your own needs. 

  2. Hall, K. D., Heymsfield, S. B., Kemnitz, J. W., Klein, S., Schoeller, D. A., & Speakman, J. R. (2012). Energy balance and its components: implications for body weight regulation. The American journal of clinical nutrition, 95(4), 989–994. https://pubmed.ncbi.nlm.nih.gov/22434603/ 

  3. The Body Weight Planner by the National Institute of Diabetes and Digestive and Kidney Diseases: https://www.niddk.nih.gov/bwp 

  4. One failing of the NIDDK model referenced in [3] is that it doesn’t seem to show this when modeling long term maintenance. When you put in a starting weight of 235 pounds, the model says you would require 3050 calories to maintain. For someone who gains 85 pounds to reach 235, it says they will maintain at 3400. No matter how long you run the simulation, it’s hard to get it to drop this maintenance level significantly below 3400. This is logically impossible. The only way someone becomes the first sort of person who maintains at 3050 is to first be a person who weighs 150 pounds and gains 85. It must be possible to transition from one to the other, although their model does not show this. Moreover, if we assume that your metabolism does eventually adapt, a 100 calorie decrease in your intake should eventually result in a loss of closer to 22 pounds. 

  5. https://stateofchildhoodobesity.org/adult-obesity/ 

  6. https://www.cdc.gov/obesity/data/adult.html 

  7. This sort of sporadic overeating could actually be a helpful adaptation. If you live in a place where food is often scarce, a rare occasion when a fatty or sugary food was available might be a good time to gorge yourself well past the point of satiation, in anticipation of lean months to come. 

  8. Hall, K. D., Sacks, G., Chandramohan, D., Chow, C. C., Wang, Y. C., Gortmaker, S. L., & Swinburn, B. A. (2011). Quantification of the effect of energy imbalance on bodyweight. Lancet (London, England), 378(9793), 826–837. https://pubmed.ncbi.nlm.nih.gov/21872751/ 

  9. A study found that 82% of people are within 10% of the BMR estimated by the Mifflin St. Jeor equation (including 75% of people with obesity), and more than half are within 5%. See https://pubmed.ncbi.nlm.nih.gov/23631843/. See also https://pubmed.ncbi.nlm.nih.gov/27184275/ for evidence that metabolic issues are not (usually) the cause of obesity. 

  10. Fildes, A., Charlton, J., Rudisill, C., Littlejohns, P., Prevost, A. T., & Gulliford, M. C. (2015). Probability of an Obese Person Attaining Normal Body Weight: Cohort Study Using Electronic Health Records. American journal of public health, 105(9), e54–e59. https://pubmed.ncbi.nlm.nih.gov/26180980/ 

  11. For example, much has been made of the supposed ability of ketogenic diets to increase metabolism. https://pubmed.ncbi.nlm.nih.gov/16389240/ discusses this and attributes to a low-carbohydrate diet that resulted in ketosis an increased expenditure of less than 100 calories a day. (This was attributed to increased thermogenesis, not any special property of ketosis.) It’s not nothing, but it indicates that most of the success of a ketogenic diet will be from calorie reduction anyway, along with potentially increased satiety. A high protein diet could also potentially provide a feeling of having more energy, resulting in an increase in physical activity. In my diet, I decided to eat plenty of clean protein for exactly these reasons, along with the desire to achieve a complete amino acid profile. I think these reasons make sense, even though I’m not counting on “ketosis” as a weight loss mechanism. 

  12. Given footnote [11], you might expect a typically high-carb vegan diet to show the opposite effect (reduced metabolism). Not so. On the contrary, a study looking at the effects of a vegan diet found that it increased the thermic effect of food by 16%, which was associated with a significant increase in weight loss compared to a control group. https://pubmed.ncbi.nlm.nih.gov/31021710/ Despite following a vegan diet myself, I don’t believe the overall results for this or any other diet show that you can “hack” the weight loss process by causing the body to run hotter, at least not significantly. In reading many papers while writing this, what struck me the most was how many of them were limited or flawed outright. I believe it is best to follow a diet that is healthy and sustainable over the long term, even if a few clinical trials would have you believe that one approach or another could net you a 10-20% faster rate of weight loss. On this note, one very large study found that following a vegetarian or high-carb diet was positively correlated with possessing a lower BMI. https://pubmed.ncbi.nlm.nih.gov/11320946/ Thus, regardless of its weight loss credentials, a high-protein diet may not be as successful at weight maintenance. 

  13. A calorie-restriction diet can only guarantee success if you actually follow it, which means accurately calculating your energy expenditure (the number of calories you burn every day) and accurately counting the number of calories you eat. Both are hard to do. My methodology gives you a quasi-guarantee of success if you can follow instructions like “eat fewer calories this week than last week”, even if you don’t know your true metabolic rate or can’t count calories accurately. 

  14. Hall K. D. (2008). What is the required energy deficit per unit weight loss?. International journal of obesity (2005), 32(3), 573–576. https://pubmed.ncbi.nlm.nih.gov/17848938/ 

  15. Ibid. 

  16. I mean this in the sense that well-situated people can have a high degree of confidence in their success. Luck is involved in the question of whether you are “well-situated” or not. Many people are not so fortunate: if you don’t have time to cook your own food, are in a “food desert”, have limited access to health care, are the victim of misinformation, or have mental health issues or an eating disorder, then you’re unlucky in the latter sense. The unfortunate fact of America’s public health crisis is that most people are in at least one of these groups. That’s why public health experts don’t see individual behavioral changes (dieting) as the solution to the problem. It’s not that relatively well-placed, high-information people can’t see success (if they approach dieting with a reasonable plan), it’s that most people don’t have that access. So when I say that dieters do have some control over whether they succeed, this doesn’t mean that a dieter who fails has only themselves to blame. Individuals are “free” to quit eating as much in much the same way as you are “free” to quit your job. It’s true, but it misses the point. 

  17. Flegal, K. M., Graubard, B. I., Williamson, D. F., & Gail, M. H. (2005). Excess deaths associated with underweight, overweight, and obesity. JAMA, 293(15), 1861–1867. https://pubmed.ncbi.nlm.nih.gov/15840860/ 

  18. Flegal, K. M., Kit, B. K., Orpana, H., & Graubard, B. I. (2013). Association of all-cause mortality with overweight and obesity using standard body mass index categories: a systematic review and meta-analysis. JAMA, 309(1), 71–82. https://pubmed.ncbi.nlm.nih.gov/23280227/ 

  19. Global BMI Mortality Collaboration, Di Angelantonio E, Bhupathiraju ShN, t al. Body-mass index and all-cause mortality: individual-participant-data meta-analysis of 239 prospective studies in four continents. Lancet. 2016;388(10046):776-786. https://pubmed.ncbi.nlm.nih.gov/27423262/ 

  20. Flegal KM, Ioannidis JPA. A meta-analysis but not a systematic review: an evaluation of the Global BMI Mortality Collaboration. J Clin Epidemiol. 2017;88:21-29. https://pubmed.ncbi.nlm.nih.gov/28435099/ 

  21. Aune, D., Sen, A., Prasad, M., Norat, T., Janszky, I., Tonstad, S.,Romundstad, P., & Vatten, L. J. (2016). BMI and all cause mortality: systematic review and non-linear dose-response meta-analysis of 230 cohort studies with 3.74 million deaths among 30.3 million participants. BMJ (Clinical research ed.), 353, i2156. https://pubmed.ncbi.nlm.nih.gov/27146380/ 

  22. Within reason, of course. If you modify the source code of PyWeight to let you target 5 pounds of weight loss per week, then the program will happily tell you that you need to eat a negative number of calories. Statistics can’t force you to start out with sane parameters, but as long as they are reasonable for someone of your height and weight (e.g. 1-2 pounds a week for most people), PyWeight will guide you onto the proper path. 

  23. Craig, W. J., Mangels, A. R., & American Dietetic Association (2009). Position of the American Dietetic Association: vegetarian diets. Journal of the American Dietetic Association, 109(7), 1266–1282. https://pubmed.ncbi.nlm.nih.gov/19562864/ 

  24. This basic strategy of eating plenty of vegetables and fruits, along with clean protein and fiber to help you feel fuller, is more or less that recommended by the UK’s National Health Service. https://assets.nhs.uk/tools/download-panels/data/weight-loss/pdf/all-weeks.pdf 

  25. Wing, R. R., & Phelan, S. (2005). Long-term weight loss maintenance. The American journal of clinical nutrition, 82(1 Suppl), 222S–225S. https://pubmed.ncbi.nlm.nih.gov/16002825/ 

Looking at the distribution of ratings on Goodreads

27 February 2022


Goodreads is a popular site for tracking books you have read and finding new ones using your ratings. Many people have noticed that Goodreads ratings have a surprisingly small variance: it seems like most popular books have ratings around ~3.8. This suggests that a book the community assesses as fairly poor and one that it says is great will be closer in rating than the descriminatory capacity of a single user (you can’t grant half-stars in your Goodreads ratings)!

Is the situation really as bad as all that? Can anything be done about it?

One user scraped Goodreads’ API fairly recently and made the data publicly available. I downloaded this data and performed some analysis on the distribution of ratings on the site.

One thing I found right away is that “only” about half of the films in the dataset have ratings between 3 and 4. There’s a bigger issue, though: about 10% of the films have a rating of precisely 4. That’s because this includes a ton of films with only a few ratings, and 4 is the most common rating on the site. What we need is a model.

What we’d like to have is a distribution function describing our knowledge about the dataset. This is usually called a Bayesian prior. Let’s consider a simplified example.

Bayesian update

If the data were in the form of thumb-ups and thumb-downs, our data would consist of a proportion of thumb-ups out of the total number of ratings. In other words, it would be binomial data. This entails that our prior distribution would be the corresponding conjugate prior, the Beta distribution. If we fit a Beta distribution to the data, we would get a function with two parameters, α and β.

These parameters have a deeper meaning. Bayesian update allows you to introduce new information to the prior knowledge you have, and tells you what your resulting beliefs about the question at hand should be. When your prior is a Beta distribution, the parameters are simply added to the new evidence. If you have a new book with x thumb-ups and y thumb-downs, the prior evidence suggests that the book’s true proportion of thumb-ups to thumb-downs is x+α : y+β. Because the resulting rating is adjusted by adding these “fake” ratings, α and β are sometimes called pseudocounts.

Alternatively, with a simple substitution, you can think of yourself as adding c = α + β fake ratings with the average value m = α / (α+β). This approach is given the name “bayesian average” by Paul Masurel, here.

We don’t have just two possibilities, of course, we have five: five different star ratings. A distribution with several options seems like it would be appropriately characterized by the generalization of the beta distribution known as the Dirichlet. This approach is what Masurel suggests:

If you have the relevant data and infinite time, you may set these two values by fitting a Dirichlet distribution on the dataset of the ratings of all your computer books. However it is very common to just choose a pair of parameter that mimick the behavior that we are looking for.

I’ve seen this approcah recommended by several others as well. Okay, I have the relevant data and infinite time, but let’s consider whether fitting a Dirichlet really makes sense here. I believe that it actually does not!

The problem is that the Dirichlet (and actually, the beta distribution) are conjugate priors for distributions of categorial variables, not discrete ones. The binomial distribution, for instance, has success and failure conditions, not 0 and 1. The multinomial distribution that has any number of distinct categories.

Why is this an issue? Well, suppose we had distinct categories. Users could describe their reaction to a book as “green”, “orange”, “red”, “yellow”, or “blue”. (Assume that users wouldn’t immediately derive a mutually agreed upon ranking system for these colors, i.e. they really are just colors.) In that case, there would be no inherent correlation between colors, and you couldn’t average them into an overall color. Star ratings are precisely the opposite. They’re numerical ratings giving a quality score, and usually what we care about is just the average.

If you fit a Dirichlet distribution to star ratings, the resulting distribution will be underconfident about the behavior of the average star rating, because it treats the ratings not as comparable numbers but as incomparable categories. More specifically, a set of star ratings will converge to their individual true values as new ratings come in more slowly than their average will converge to its true value.

As proof of this, here is the result I get when fitting the Dirichlet distribution to the Goodreads data.1 I am plotting the Dirichlet reparameterized to show the strength of the prior on the average value.

plot of average values in the Goodreads data vs the prior derived from the
Dirichlet model

As you can clearly see, the result is underconfident. Don’t use the Dirichlet as a prior for star ratings, especially when you only care about the average!

What should you use instead? I think the beta distribution is actually more reasonable here. Instead of modeling the proportion of successes and failures in a series of Bernoulli trials, we can think of it as modeling the average value of a series of individual equally weighted fractional successes, where e.g. 5 stars means perfect success and 1 means complete failure.

If anyone knows of a better way of motivating the use of a particular distribution here, please do contact me by email or on Github and let me know.

The results

I filtered out books with fewer than 50 ratings, to eliminate books for which no rating consensus has yet been formed. (I also implemented some special handling for books that have at least one star rating which no one has yet given them, though this is a technical detail which I’ll let the interested read the code to see.)

plot of average values in the Goodreads data with a fitted beta
distribution

The beta distribution appears to be a very good fit for the data. Experimentally, the distribution of dirichlet averages seems to become more and more beta-like as the strength of the prior increases.

The smoothed and filtered values show the Goodreads ratings to be worse (less informative) than we initially feared. Fully 80% of adjusted ratings are between 3.5 and 4.2, a range considerably less than the one star’s worth of discrimination available to individual users when rating a book. Virtually every book on the platform that has enough ratings to be reliable gets at least a 3, and virtually none gets more than a 4.5.

A partial solution

Given that we have a model of the data, we can adjust the average rating of books with too few ratings to give ourselves a more accurate estimate of the true average of the book. Furthermore, we can use the model to determine what percentile a given rating corresponds to: my analysis suggests that a book with a Bayesian rating of 4 is better rated than 2/3 of books on Goodreads.

For those who are interested in more accurate ratings (for books with a small number of them) and the ability to see a book’s percentile, I have made a user script which does just that!

Does the ability to see rankings instead of a raw score solve the problem with Goodreads? It certainly makes things better, but in my opinion it doesn’t solve the basic problem, which is that Goodread ratings just kinda suck. They’re inconsistent, prone to audience biases, and fairly arbitrary.

A sampling:

As it happens, I think all of these works are great. (Books I hated are similarly all over the place.) The problem is not just that the ratings are so widely varied, it’s that there’s no apparent rhyme or reason to the order these books are in. When a film I think is great has a relatively low rating on IMDb (example: I’m Thinking of Ending Things), there is usually an obvious, comprehensible reason why most people didn’t like (or were indifferent to) the film. Likewise, I thought Star Trek Into Darkness was quite bad, but I understand why most people liked it. Nothing of the sort can be said for Goodreads: not many people dislike Shakespeare — so what’s happening here? Unlike with IMDb, where the ratings (though clearly inflated, with most movies between a six and an eight) do frequently indicate something about the quality of the movie, I’ve found no way to extract the same value from ratings on Goodreads.


  1. Here, I’m eliding how we arrive at the precise α and β values: the Dirichlet is parametized by a vector α that behaves exactly the same way as the α and β values of the beta distribution. Since our data has 5 possibilities, α will contain 5 values. To update a Dirichlet prior, each of the values is added to the corresponding value in the evidence vector. As Masurel shows, since we care only about the average value we can simplify the resulting α vector down to two variables; the sum of the vector, and the average (number of stars) that a book with exactly α reviews would have. These values are just the c and m of the bayesian average approach. 

How I Defeated Wordle with Python

05 January 2022


Wordle is a simple online word game. It is played by similar rules to the classic game Mastermind, which is known in the United States through the version licensed to Hasbro. Wordle is played with words instead of arbitrary configurations of pegs or numbers, and there is one added twist: each of your guesses must also be a word in Wordle’s dictionary, not any arbitrary configuration of letters.

For those unfamiliar with Mastermind, I recommend clicking through to try the game or reading this brief description of the rules: every day, the game randomly chooses a five letter word from a predetermined list of possible words. On each of your turns, you will guess a word that must be in a dictionary contained by the game. If you guess the word within 6 turns, you win. The game tracks your streak of wins and the number of guesses it takes you to solve each one.

For each guess, the game will tell you whether each letter

Each letter in the guess has a 1-1 correspondence with the same letter in the answer, if there is one. This means that if the correct answer is “means”, and you guess “green”, you would see the following:

the word "green" with the first "e" and the "n" highlighted in yellow

The yellow color indicates that the chosen letter appears in the solution, but that it is in the wrong place in our guess. The second “e” appears in grey, because there is no second “e” in the solution. There must be a 1-1 correspondence between each guessed letter and a solution letter, and if they were both highlighted in yellow, the “e” in the solution would correspond to two letters in the guess.

Likewise, a green letter, indicating a letter in the correct location, takes precedence over a letter in the incorrect location. So if the solution was “rules”, we would see the following:

the word "green" with the "r" highlighted in yellow, and the second "e" highlighted in green

Here as before only one “e” is highlighted, but the second is in green rather than the first being in yellow, since the letter in a correct location takes precedence over a letter in an incorrect location.

I’d like to solve the puzzle

When I come across a puzzle like this, I’m immediately compelled to think about optimal ways to play it. For example, the player is in the same knowledge position at the beginning of the game every time. Since the opponent is not adversarial (a random 5 letter word is picked, not a word intended to be problematic for any particular guessing strategy), the optimal strategy necessarily means guessing the same word first every time. Few human players likely play this way.

In fact, given that the solution word is drawn from an unchanging list of words before the game begins, this means you can just pre-calculate the optimal guess given each previous guess and response from the game, laying out every possible game state in a tree-shaped diagram.

I have written a solver for Wordle in Python which searches for an optimal (per several rules, see below) game tree, and saves it in a JSON file, which turns out to only contain about 15 KB when compressed. The optimal game tree for Wordle therefore turns out to be pretty small!

How the solver works

The ideal game tree would have 3 constraints:

I have made several simplifications in order to quickly get a reasonable solution:

A constraint to keep the maximum path under seven guesses turns out not to be necessary, because a solver with just the two constraints above will never take more than five.

As it turns out, the game actually has two word lists: one is the set of words it will consider using for solutions, just over 2000 words. The other is the full dictionary of words it will accept as guesses. I found both these lists in the Javascript source code of the game. Clever use of the full list of guessable words would in same cases allow faster solves, but my solver is so efficient even without this that I haven’t seen fit to implement it yet. So my program only guesses words that could, in theory, be used by the program as solutions.

Likewise, always guessing the word that will result in (on average) the smallest number of possible solutions is only an approximation of optimal guessing. There are 2314^6 = 153525361154699100736 different possible routes through the game, and so brute forcing your way to an optimal solution is (while not unthinkable as it is in chess), probably unworkable in Python. Clever pruning of the search tree should help (and I’m going to look at this at some point), but is not as easy as it is in games with an opponent. In chess, you can simplify the tree by assuming that the opponent will always make the move that is worst for you. In Wordle, because the solution is random, you have to always consider all possibilities and try to find the guess that is optimal on average.

Still, though, reducing the average number of live solutions as much as possible is a very good approximation of ideal play. Consider the following situation:

A series of Wordle guesses: "raise", "fiend", "pyymy", followed by the correct solution, "tiger"

After the second guess, the computer makes the interesting choice “pygmy”. This may seem counter-intuitive. Not only do we already have two vowels confirmed to be in the word, “i” and “e”, the guess actually contains “y” twice, which reduces the number of letters contained in the guess. As a matter of fact however, this guess is rather astute. The computer will always find the solution on the fourth guess depending on the outcome of this query.

Using ‘A’ to mean absent, ‘P’ to mean present, and ‘C’ to mean correct:

AAAAA → liver
AAAPA → timer
AAPAA → giver
AACAA → tiger
PAAAA → viper
CAAAA → piper

Being in a point in the game tree where there are six descendent guesses means that there are six possible solutions. Clearly, optimal behavior would be to always find the solution on the next turn. A brute force search would find this solution, but so would the heuristic of eliminating as many possibilities as possible. With “pygmy”, we always eliminate five possible solutions, the best result possible in this case.

Using a heuristic like this is much faster than a brute force search, because it generates a guess in each situation without needing any recursion at all.

Results

This was just a quick little project for me, taking a few hours, so the code is relatively unoptimized. Determining the best first guess requires searching every combination of possible guess and possible solution, and this takes several hours. The complete tree is generated in only a few minutes after that. Because it is stored as JSON, a player for the game is included that doesn’t have to do any searching - it simply reads the next guess out of the game tree.

Stats:

The solution is found, on average, in 3.51 guesses. In the worst case scenario, the solution is found is 5 guesses. The histogram of outcomes is as follows:

1: 1 (0.0%)
2: 65 (2.8%)
3: 1085 (46.9%)
4: 1074 (46.4%)
5: 90 (3.9%)

If anyone improves on my solution by utilizing the complete dictionary or achieving a brute force solve of the game, I would be curious to hear how much you manage to improve on these statistics.

Updates

Jan 11: I discovered that the program was treating all guesses that resulted in the same average reduction to the live possibilities as equivalent. We can, without violating the constraints given above, choose to prefer guesses that are also possible solutions to the current puzzle. Adopting this change reduced the average solution distance from 3.68 to 3.51 guesses.

Code

All of my code for this project is open sourced, under a GPL3 license, on Github. This includes:

All the Python code should run on recent versions of Python 3, including the Pypy implementation for additional speed.

Solving the vehicle routing problem for research biologists

20 June 2021


Code accompanying this blog post is available on Github.

My partner is a research scientist working in a lab group. Every year, members of the lab are expected to go on long driving trips to collect specimens for research. This labor is shared among the members of the lab (in practice, the graduate student members of the lab). So even though one lab member’s work might only focus on a single species, a research trip might collect many different species, even if the lab member who requires them is not present on the trip.

In practice, the planned destinations are known at the beginning of the academic quarter. The trips need to be planned out in such a way that effort and route duplication is minimized, while each individual only goes on as many trips as their workload will bear.

While not a professional programmer or computer scientist by any means, I’ve been tinkering with software long enough that coming up with ways to solve problems like this is not that hard for me, and so I’ve been asked on several occasions now to solve the routing problem for them. Having done that, I thought I’d document my process (using entirely open source software and data) and provide a few hacked together scripts to simplify the process.

The core piece of the puzzle is Vroom, a software project explicitly designed to solve the vehicle routing problem in an efficient way. It’s designed to integrate with OSRM, a routing engine built with OpenStreetMap data. If you’re curious about the system, I recommend this underappreciated talk about Vroom.

Dealing with constraints can be complicated. Fortunately, I never had to worry about blackout dates, because the routes once calculated could begin whenever the driver was ready to start. That said, Vroom has one significant limitation: the only metric it allows you to optimize is total drive time. If there is 3 weeks worth of work to do, and it minimizes total drive time to send one vehicle on a very long trip, Vroom will do that. You can’t tell Vroom to, for example, minimize the length of the longest trip, or to make all trips as close together in length as possible. I thought at first that this would make Vroom unusable for my tasks (it certainly doesn’t seem like it would work for hot pizza delivery, for example), but with a lot of handheld tinkering you can get it to work using availability constraints on the vehicles. I also didn’t find any software that would handle the problem better.

While OSRM does provide a very generous free API, it’s still too limited for the kind of routing I needed to do. Vroom works by requesting a routing matrix from OSRM that contains the travel times between every pair of points you submit. Sometimes I would be given a thousand or more destinations. This meant I had to run a routing server myself. Fortunately I have an always-on server that was more than up to the task.

You can find the routing server at osrm-backend. There’s a Docker image, but as I prefer to run software in different containers I simply built it myself. The map data I needed (the US West subregion) is available to download from Geofabrik. After pre-processing, the data for the western United States is “only” 14 GB.

Very clear instructions for setting up the software are available on the project’s wiki page on Github. You’ll probably want to use the MLD pipeline as the pre-processing step is much faster. The page recommends the CH pipeline specifically for large distance matrices (exactly what we have), but as we’re only going to submit a small number of requests to this server in its lifetime, we can afford for them to be slightly slower.

Vroom has a documented JSON format that describes the available vehicles, the destinations they need to reach, and other constraints. I wrote a script to create the necessary input for Vroom based on data in a spreadsheet (exported to a TSV file).

The format of this file is

Latitude <tab> Longitude <tab> Skills <tab> Time

Skills is probably the most interesting feature I made available. Sometimes one scientist is particular good at identifying one species, and so the plan is to make sure they’re in the vehicle for every destination that involves collecting that species. In the spreadsheet, the scientists can put a number (or several numbers separated by a comma) in the corresponding Skills cell. Each vehicle has a corresponding list of numbers which identify the scientists in each vehicle (a scientist can go on multiple trips and so be in multiple vehicles). Vroom will use the skills feature to only send a vehicle to a destination if there is a scientist in the vehicle who can identify the specimen there.

The Time field is occasionally useful too. It’s an estimation of how long the vehicle will spend at the destination not moving (in seconds). In most cases, the scientists I was routing for would simply park the car, clip a branch of a tree or shrub, and get back in the car, so this was effectively zero. In some cases, however, the GPS point could be a mile or more from the road, meaning that a longer hike was in order.

The scripts that I’m making available are very much hacked together, not up to my usual standard for code I make public. For example, there are important constants in the makejson.py file that have to be set or adjusted when the script is run. I didn’t come up with a convenient way to use argparse for this.

We have the already-discussed list of VEHICLES:

VEHICLES = [[1,2],[3,4]]

This identifies two vehicles / trips, where two scientists (numbered 1 and 2) are in the first, while two other scientists (numbered 3 and 4) are in the second.

We have lists of starting coordinates and ending coordinates for each vehicle. These are in [longitude, latitude] format because that’s what Vroom itself uses.

STARTCOORDS = [[-118.44, 34.068], [-118.44, 34.068]]
ENDCOORDS = [[-118.44, 34.068], [-118.44, 34.068]]

Note how the starting and ending coordinates for the vehicles are all the same in this case. That’s a common pattern (starting and ending at the university), but you’re not locked into doing that by any means.

Last we have TIMEAVAIL:

TIMEAVAIL = [400000, 400000] # seconds

This is the messiest part. It’s the total drive time allowed for each car. In theory, this would be perfectly known to the participants in advance (they could allow more time for stops, breaks, sleeping, etc., as needed). In practice, though, it’s a balancing act. The research trips have to be done, so even if finishing all of them would take more time than the scientists theoretically have, put together, that just means that they have to find more time from somewhere. So in practice the only workable approach that I found was setting this value unreasonably high for each car, then reducing it selectively to get routes that looked reasonable. When you can’t reduce it any more without the router failing, you have a roughly optimal solution.

Vroom will print a large JSON file with the results (and the routes themselves from OSRM in polyline format), and so I have another script that takes the result and prints the result as a long list of coordinates for each vehicle, and also exports the routes (as lists of coordinates) to text files, so that they can be shown on a map. Theoretically I ought to create KMLs or something more useful, but this turned out to be more complicated than I had hoped to do in Python and so I ran out of time.

My repository contains a run.sh file that shows how I run the program for my specific setup. Obviously you’ll need all the pieces in place (Vroom, OSRM on a server, etc) in place for this to work. Running the script, even for a large list of destinations, takes only 3-5 seconds, so re-running it while changing the parameters is no big deal.

The results are extremely nice looking:

A map showing a solution to the vehicle routing problem in 
California

Thanks to GPS Visualizer for making it easy to plot these routes.

This solution shows three different routes, constrained to a limit of about a week per trip. Several of the points have additional constraints, such that only one scientist can collect the sample, and this has still allowed for a nice solution. In many cases, it’s possible for human intuition to come up with a map that looks a lot like this one, but getting the exact order of coordinates right turns out to be crucial in saving hours of drive time.

At the end of the day, I suspect my exact approach won’t be completely workable for most people. You’ll have slightly different constraints than me, a different input format for your data, or some other problem. That’s why I’m just sticking these scripts online instead of spending the time to make sure they’re as functional and well-written as they could be. Hopefully, others will be able to look to them for help in solving their own routing problems.


Next page
©2022 Adam Fontenot. Licensed under CC BY-SA. About Me Projects RSS Feed