Flesch reading ease for stylometry?

The Flesch reading-ease score (FRES, also called FRE – ‘Flesch Reading Ease’) is still a popular measurement for the readability of texts, despite some criticism and suggestions for improvement since it was first proposed by Rudolf Flesch in 1948. (I’ve never read his original paper, though; all my information is taken from Wikipedia.) On a scale from 0 to 100, it indicates how difficult it is to understand a given text based on sentence length and word length, with a low score meaning difficult to read and a high score meaning easy to read.

Sentence length and word length are also popular factors in stylometry, the idea here being that some authors (or, generally speaking, kinds of text) prefer longer sentences and/or words while others prefer shorter ones. Thus such scores based on sentence length and word length might serve as an indicator of how similar two given texts are. In fact, FRES is used in actual stylometry, albeit only as one factor among many (e.g. in Brennan, Afroz and Greenstadt 2012 (PDF)). Over other stylometric indicators, FRES would have the added benefit that it actually says something in itself about the text, rather than being merely a number that only means something in relation to another.

The original FRES formula was developed for English and has been modified for other languages. In the last few stylometry blogposts here, the examples were taken from Japanese manga, but FRES is not well suited for Japanese. The main reason is that syllables don’t play much of a role in Japanese readability. More important factors are the number of characters and the ratio of kanji, as the number of syllables per character varies. A two-kanji compound, for instance, can have fewer syllables than a single-kanji word (e.g. 部長 bu‧chō ‘head of department’ vs. 力 chi‧ka‧ra ‘power’). Therefore, we’re going to use our old English-language X-Men examples from 2017 again.

The comics in question are: Astonishing X-Men #1 (1995) written by Scott Lobdell, Ultimate X-Men #1 (2001) written by Mark Millar, and Civil War: X-Men #1 (2006) written by David Hine. Looking at just the opening sequence of each comic (see the previous X-Men post for some images), we get the following sentence / word / syllable counts:

  • AXM: 3 sentences, 68 words, 100 syllables.
  • UXM: 6 sentences, 82 words, 148 syllables.
  • CW:XM: 7 sentences, 79 words, 114 syllables.

We don’t even need to use Flesch’s formula to get an idea of the readability differences: the sentences in AXM are really long and those in CW:XM are much shorter. As for word length, UXM stands out with rather long words such as “unconstitutional”, which is reflected in the high ratio of syllables per word.

Applying the formula (cf. Wikipedia), we get the following FRESs:

  • AXM: 59.4
  • UXM: 40.3
  • CW:XM: 73.3

Who would have thought that! It looks like UXM (or at least the selected portion) is harder to read than AXM – a FRES of 40.3 is already ‘College’ level according to Flesch’s table.

But how do these numbers help us if we’re interested in stylometric similarity? All three texts are written by different writers. So far we could only say (again – based on a insufficiently sized sample) that Hine’s writing style is closer to Lobdell’s than to Millar’s. The ultimate test for a stylometric indicator would be to take an additional example text that is written by one of the three authors, and see if its FRES is close to the one from the same author’s X-Men text.

Our 4th example will be the rather randomly selected Nemesis by Millar (2010, art by Steve McNiven) from which we’ll also take all text from the first few panels.

3 panels from Nemesis by Mark Millar and Steve McNiven

Part of the opening scene from Nemesis.

These are the numbers for the selected text fragment from Nemesis:

  • 8 sentences, 68 words, 88 syllables.
  • This translates to a FRES of 88.7!

In other words, Nemesis and UXM, the two comics written by Millar, appear to be the most dissimilar of the four! However, that was to be expected. Millar would be a poor writer if he always applied the same style to each character in each scene. And the two selected scenes are very different: a TV news report in UXM in contrast to a dialogue (or perhaps more like the typical villain’s monologue) in Nemesis.

Interestingly, there is a TV news report scene in Nemesis too (Part 3, p. 3). Wouldn’t that make for a more suitable comparison?

Here are the numbers for this TV scene which I’ll call N2:

  • 4 sentences, 81 words, 146 syllables.
  • FRES: 33.8

Now this looks more like Millar’s writing from UXM: the difference between the two scores is so small (6.5) that they can be said to be almost identical.

Still, we haven’t really proven anything yet. One possible interpretation of the scores is that the ~30-40 range is simply the usual range for this type of text, i.e. TV news reports. So perhaps these scores are not specific to Millar (or even to comics). One would have to look at similar scenes by Lobdell, Hine and/or other writers to verify that, and ideally also at real-world news transcripts.

On the other hand, one thing has worked well: two texts that we had intuitively identified as similar – UXM and N2 – indeed showed similar Flesch scores. That means FRES is not only a measurement of readability but also of stylometric similarity – albeit a rather crude one which is, as always, best used in combination with other metrics.

Top 10 words from Frederik L. Schodt

Cover of Frederik L. Schodt's Dreamland JapanOut of the many authors who publish on comics, Frederik L. Schodt is one of the few with a truly distinct writing style – neither academic nor fannish, neither highbrow nor colloquial, his writings are full of rather obscure words, some of which I have never seen anywhere else. Recently I re-read the beginning of his book Dreamland Japan, and while doing so, just for fun,* assembled this list of my favourite eccentric words therein and their meanings (as far as I could find out):

to accord – p. 19: “Japan is the first nation in the world to accord ‘comic books’ […] nearly the same social status as novels and films.” – to grant, to give.

bone-crushing – p. 28: “Yet along with this celebration of the ordinary is the bone-crushing reality that the vast majority of manga border on trash.” – back-breaking, depressing (cf. German: ‘erdrückend’).

hari-kari – p. 11: “in due time both words [manga and anime] will undoubtedly be listed in the standard English dictionary along with other Japanese imports like ‘hari-kari’ and ‘karaoke.'” – variant of harakiri (ritual suicide).

finicky – p. 13: “In Japan, people’s names are usually listed with the family name first and the given name last. Certain academic types in the English-speaking world are rather finicky about this convention and insist on preserving it even in English texts” – difficult to please, demanding.

to flounder – p. 34: “Japanese people have floundered about trying to the right term to describe the sequential picture-panels that tell a story.” – to struggle.

full-figured – p. 26: “Japanese manga offer far more visual diversity than mainstream American comics, which […] still reveal an obsession with muscled males and full-figured females” – according to Wiktionary, ‘full-figured’ means ‘fat’ or ‘plump’, but here it’s probably used in the sense of ‘curvaceous’ or ‘voluptuous’.

persnickety – p. 14: “Fans of Japanese manga (even more than academics) can be a rather persnickety and unforgiving lot” – see finicky.

profuse – p. 15: “Profuse thanks are offered to all who helped.” – plenty, abundant.

raga-like – p. 14: “[…] with raga-like stories that may continue for thousands of pages” – maybe Schodt means, ‘as lengthy as an Indian epic (raga)’?

satori-like – p. 21: “his face lit up in a satori-like realization” – (Buddhist) enlightenment.

* On the other hand, this little exercise can also be seen as a tentative reflection on the serious topic of academic writing style and the way in which we, as scholars, communicate our findings.