Fantastic that they share this information and an enjoyable read.

This said, users can’t currently order AirBNB search results by price ascending or descending.

https://community.withairbnb.com/t5/Hosting/Sorting-listing-...

I think we should remember to empower the humans a bit more than the machines?

The headline screams red flag to me. Calling something "machine learning-powered" isn't a guarantee of improved performance at all. Their baseline is already a fairly complex model, so there's no way to gauge how useful having a model at all is.

The partial dependency plots they show have gigantically wide error bars on them - they talk as if they offer a really nice confirmation of intuition, but the real relationship could be anywhere in those intervals, so it's probably weak evidence at best. The only one of the three where you could say with confidence that there's a relationship is the higher price -> lower score one. Add it all up, and you've discovered that people prefer cheaper stuff, with maybe a whisper of it being more complicated than that. In which case, their stage 1 model likely amounts to "show them the cheap stuff first", without the developers involved realising it. This is one of the problems we have with modelling - the model could be doing something simpler than we realise.

This gets more worrisome when you consider the fact that they're doing A/B testing to check for performance improvements. How are they able to claim that they improved bookings by 13% with their stage 1 model? Improved relative to what? More importantly, how do they know that this isn't a dataset-specific effect given that the data used for the stage 1 model is "small"?

I didn't continue reading after this, the whole article looks like confirmation bias to me.

The A/B test was against a random ranking.

I find the searches on Airbnb to be pretty ineffective. Often times I’m trying to find a listing I was previously viewing and it’s nearly impossible to find it (referring only to the cases where I eventually do find it, not when it’s no longer available)

It seems to be working as designed:

"experiences that were clicked but not booked (which we treated as negative labels)"

Too bad that they don't provide any reasoning for this labelling. Also it isn't clear to me over which time span they were looking at individuals to make the label decision, i.e. are they considering sessions or just single clicks with subsequent bookings?

Related: https://blog.acolyer.org/2019/10/09/applying-deep-learning-t...

Look at the authors. China is really going to put the screws to the US and it’s going to be sooner rather than later.