Categories
science

Discontinuity tells

A particular pattern in the numbers, a misbehaviour which shows that the scale is more than just a record:

Source: Reddit Distribution of results of the Matura (high school exit exam) in Poland in 2013. The minimum score to pass is 30%. Commentary: Unusual distributions of scores on final high-school exams in Poland.

But let’s be clear — this is not about Poland. The x-axis pretends to innocence, just a passive measure of how the data fall. This pattern, though, the cratering before the threshold and then the pile-up just after, show that the threshold itself has a distorting effect.

Here’s data on the New York high school exit exam

Source: Dee, T. S., Dobbie, W., Jacob, B. A., & Rockoff, J. (2019). The causes and consequences of test score manipulation: Evidence from the New York regents examinations. American Economic Journal: Applied Economics, 11(3), 382-423. (NBER working paper 22165).

There is some discretion in the exam marking. The grade matters to what happens to the students (or the schools and teachers which produce the students), so individual data points near the threshold “sort themselves” over the line.

Here’s a non-exam example, from scoring eligibility for a social programme in Colombia. The threshold to receive benefits was 47:

Source: Camacho, A., & Conover, E. (2011). Manipulation of Social Program Eligibility. American Economic Journal: Economic Policy, 3(2), 41–65. DOI: 10.1257/pol.3.2.41

Interesting because it is an example of the whole population data having a particular “tell”: there’s a story around how points fall near these thresholds.

Also, any regression discontinuity design which assumes random assignment is going to be confounded by such near-threshold sorting. The check is to look at the count density around the threshold (and cross your fingers for no cratering/piling up).

Categories
Me technical notes

Follow on the fediverse (again)

You can follow this blog on the fediverse. Technical issues mean I may have purged my followers, so if you already followed you may need to re-follow.

On something like mastodon you can search for the blog profile @notes, clickable as https://idiolect.org.uk/notes/@notes

Previously: ActivityPub on this blog (2024-09-16)

Categories
science technical notes

Actual personalisation

I’ve been making a website which tracks metascience relevant seminars — the Global metascience seminar monitor.

The site subscribes to emails, scrapes seminar webpages (takes RSS), updates daily, takes updates and corrections by email automatically and publishes a calendar feed so you can see events in your calendar automatically.

It’s half a useful tool to keep up with an interesting and evolving area, and half an experiment in using AI. I vibe coded the website, using my openclaw agent running Claude, but the site itself runs entirely without commercial AI (it uses some local models to do ingest incoming info and decide if it is relevant) — the motivation was to avoid building in dependency on a subscription to any AI company.

The monitor is a function which I think has wide relevance to our age of information overwhelm, and to the specific capacities of AI. The AI may not be very smart, but it has practically infinite bandwidth and practically infinite patience. Rather than replace the things I am good at and might enjoy (going to seminars and learning from them, is one example), I can use the AI to do something I am relatively bad at (keeping up with 1000s of emails about seminars, picking out items I might be interested in).

The monitor also illustrates another angle of the current AI moment — true personalisation. There was a lot of talk about personalisation during the early digital era, including talk about personalised advertising. Digital technology is could be infinitely customisable, true. Surveillance companies like Meta have a vast amount of data on each of us, also true. But the digital technology of the last 20 years has not been personalised, and what is possible with AI has shown that up. Targetted advertising was mostly a con to sell to the tech companies’ true customers (advertisers), while the apps and websites we ended up with provided lacklustre personalisation at best, tending to push us all towards a homogeneous experience with a thin layer of different content on top.

(The events feed on Facebook is a particular grievance of mine. There was a point at which nearly every event had a Facebook page, nearly every friend of mine told Facebook what they were doing and what they liked, and I told Facebook what I was doing and what I liked, and yet the ‘events you might like’ suggestions were consistently mediocre. The fact that Facebook couldn’t even make a modest effort at pretending to meet this need shows clearly how their priority was never personalisation for the user).

With AI you can ask for content based on your preferences, and also build bespoke software based on those preferences. I wanted a metascience seminar monitor, and AI meant I could build it (I could, maybe, have built it without AI, but it would have taken six months). I have a whole host of personalised projects, from a text-editor which integrates with my notes, to a system which allows me to file expense claims by sending the receipt to the agent on WhatsApp.

It’s interesting to think about the things AI is particularly suited for – things where I don’t have the bandwidth, things where I don’t have the persistence, things which are annoying (like filling in forms), things which I might forget…And things where the product which is served for everyone else isn’t exactly what I want for my purposes.

The recent history of tech is one of enshittification, and the history of everyone is more admin, more forms, and a world increasingly filled with impersonal, infuriating, bureaurocracy. There are lots of reasons to hate or fear AI, but if it lets us subvert or avoid some of the homogenising, flattening, and constraining tendencies of the digital world, and make things which are particular to what we, as individuals what and need, then that’s a good thing.