TinyGem.org - bookmarking and content recommendations for people who love to read Hacker News.

T-Mobile begins blocking iPhone users from enabling iCloud Private Relay in US | Hacker News

I would think Apple has some leverage to force it if they really wanted.

If Apple really wanted to force the issue, they could tell T-Mobile no more iPhone contracts unless you do it. Apple can survive and thrive on fewer networks - the iPhone was AT&T exclusive for a long time at the beginning.

If that happened, there would be no way for T-Mobile to get a supply of iPhones. People would need to buy iPhones from Apple and then replace the SIM cards themselves. It would make T-Mobile bend pretty quickly unless they managed to get Verizon and AT&T to join them on the issue.

But then Apple has a second card to play, and that's the court of public opinion. If Apple wanted to make a public ad lambasting the carriers for undermining people's privacy, the damage would also force them to bend.

Finally, of course, there's the fact that carriers need Apple just as much as Apple needs carriers. However, between the carriers and Apple, who has $200 billion in the bank to do things themselves if they wanted?

Edit: Heck, T-Mobile has a market value of $130 billion. AT&T has a market cap of $188B, and Verizon $223 billion. If Verizon and AT&T joined T-Mobile in protest, Apple could theoretically attempt (or at least threaten) a hostile takeover of any of them. That would cause a lot of discussion among the carriers and send a strong message very quickly.

Apple could simply reimplement APN and iCloud Files and etc over their own Private Relay connections, so that any carrier who blocks Private Relay ends up blocking all push notifications to all Apple devices. They have not yet done so, but if they do, either all US carriers will have to collude to block it — which could reasonably be considered anti-competitive behavior and would attract immediate governmental review — or they'd all have to allow Private Relay, as otherwise they'd bleed millions of customers due to being hostile to iPhone.

It seems safe to assume that Apple doesn't like user APN traffic being tracked any more than the rest of their traffic, and so I would imagine this is essentially the roadmap for all Apple Services provided to their users over the next few years.

It's also possible the Beta is just to see what happens when they make the traffic visible and possible to block, and having seen the carriers tending towards "block anything that inhibits tracking", they might simply target Private Relay 2.0 as "HTTP3 over 443/udp to http3.apple.com" and have that be an N-way reflector point for all customer-provided Apple services, including Private Relay and the rest as different channels over that multipath HTTP3 layer. They chose to implement Private Relay in a way that could be detected, and are measuring how neutrality-hating carriers respond. Assuming it's their final answer is not a bet I'd take.

It seems like any kind of forceful action would trigger government review. The safest option to me seems to be just having a modal pop up saying "T-Mobile blocks private relay from being used".

It isn't a forceful use of power by Apple and it clearly informs just the simple truth that the users carrier is hostile and maybe they want to switch.

In the US, big Carrier A implements the full array of iPhone capabilities for carriers under Settings > Cellular; big Carrier B does not. Most people aren’t aware of what they’re missing, but for example, when implemented by the carrier properly, the cellular data usage heading is “this billing cycle” instead of “however long it’s been since you last clicked Reset Statistics manually”; and Settings > Phone > Call Forwarding has an iOS native UI rather than forcing you to dial 1980s-style *#12##0112636382* modem strings into the dialer.

Does your carrier implement this? If not, will you switch carriers? It’s unclear what degree of importance you assign to your expectation, as it could be anything from “for the tasks I care about” to “in every possibly way without exception”. (For me: It’s not important eniugh to switch carriers on its own, though it’s certainly a factor if I have greater reasons.)

That has never been the case. Carriers have been tampering with web traffic for a while now. If T-Mobile is specifically tampering with the network, the problem is external and out of the control of Apple.

> If Apple wanted to make a public ad lambasting the carriers for undermining people's privacy, the damage would also force them to bend.

That ad would be candy to the Apple PR team trying to push the “Apple is secure and respects your privacy” campaign. I bet we’ll see Apple use the court of public opinion here, and win with it.

I'm trying to think of a case in which Apple has used public opinion in this way. The closest I can come up with is Adobe Flash, but Apple was the one blocking a product on its platform then.

If Apple escalates this into a dirt-flinging war, I don't think any domestic carriers would take offense at reminding the public that Apple is the only one among them that still does business with China. But neither one will escalate things, because both Apple and every US cell carrier have so many skeletons in their closet that trying to call one another out wouldn't just be hypocritical, it would be mutually assured destruction.

I don't think that would be effective, everyone already knows that Apple builds their phones in China, it says on the back. The fact that your cell carrier wants so badly to spy on you that they are willing to go to go to bat with Apple will, however, surprise some people.

Nah - that wouldn't work. Apple would just point out that they use networking gear made in China. Brilliant.

My point is that carriers and manufacturers have so much dirt on each other that trying to escalate things would just hurt them both. The reason why Apple (and mobile carriers, for that matter) don't take swings at each other is because they both need the other to look as pristine as possible to sell units. They have a mutual interest in looking good together, and neither Apple nor the carriers have any vested interest in breaking that relationship.

I don't think the American public would particularly care – and some would probably even support – that Apple does business with China. If that's the best the carriers can throw at Apple, versus Apple cutting them off from the single device doing the heaviest lifting to keep them relevant, then yikes.

Oh, that's certainly not the worst they'd grab for, but more of an example where they can call their bluff. Cell carriers and hardware manufacturers alike get bent over backwards for compliance in the United States, trying to assert that you're "the private one" is just going to get you called on every other front. It's not even a question that these companies do shady things, the real question is more about the lengths they'd go to diminish their competition.

Again though, rupturing this conversation is mutually assured destruction. The reason why Apple won't call T-Mobile's bluff is because it's better for them to look like a symbiotic company than an adversarial one, and T-Mobile can get away with this because data protection in the US is a moot-point anyways. It's about as unremarkable as news gets.

Hell, Apple was even nice enough to give T-Mobile a special error message when you try to use Private Relay:

> "Your cellular plan doesn’t support iCloud Private Relay. With Private Relay turned off, this network can monitor your internet activity, and your IP address is not hidden from known trackers or websites."

I wouldn't call it security theater if I couldn't see the curtains on the left and right.

It's strange that you think Americans would equate "people in China are suffering" and "X is spying on me".

Your suggested response doesn't change my mind at all, it seems quite desperate and pathetic. And I want to see a stronger moral stance on trade with China.

> Apple could theoretically attempt (or at least threaten) a hostile takeover of any of them.

If I were any of the carriers, I wouldn't worry about this in the slightest.

Apple attempting to gain ownership of a mobile carrier in order to impose it's will on the market would be met with incredibly harsh regulatory scrutiny.

Beyond that, there's a strategic reason Apple hasn't launched their own mobile offering. The minute Apple owns a particular mobile carrier, they would be pretty well cut off from the other mobile carriers, or they would have to negotiate deals that would probably be argued to be collusive trade practices.

The real solution is that the United States needs real data security and privacy laws that prevent network operators from reselling your usage history, location tracking, and other personal details. It's a national security issue at this point.

I was thinking the same, then I remembered that Google has its own phones, it owns Android, and it managed to start its own cellular network without a whisper. So, no reason why Apple can't do the same.

Google is in slightly more of a "home-grown" situation, but more importantly Lina Khan was not in her current chair back when all that stuff happened. In marked contrast to previous officeholders, she won't be deferring to the "monopoly is good" FCC.

> Heck, T-Mobile has a market value of $130 billion. AT&T has a market cap of $188B, and Verizon $223 billion. If Verizon and AT&T joined T-Mobile in protest, Apple could theoretically attempt (or at least threaten) a hostile takeover of any of them.

OTOH, one way to become the most valuable company in history is to not go pulling stunts like that. Nothing the street loves more than predictability.

Of course, the odds of this are extremely small. It's just more to show that Apple has more leverage than the carriers in this situation.

Edit: Another, "smarter" tactic that Apple might use is by sending messages to the Board of Directors. If Apple can get the Board of Directors on their side (or at least convince them that management is fighting a war they can't win)... another way to freak out execs at the carriers.

The odds of any American company could start scooping up cell carriers without reproach is not just "small", but more along the lines of "complete impossibility". The SEC already gives Apple the stink-eye for gobbling up C-lister startup companies; if they tried acquiring anyone in the S&P 500, every trade commission in the world would be on them within seconds.

I also think it's silly to equate a company's power to the amount of money they have (at least in the first world) but your hypothetical does raise an interesting question: who's deeper in bed with the State, Big Telecom or FAANG? All of them answer to the government, even T-Mobile; but who's got the most favor? Understanding the heinous stuff the American government got away with when they had telecom under their thumb doesn't set a very optimistic baseline of expectations. It might even lead certain people to believe (surprise surprise) that Apple's dedication to privacy doesn't really mean much when there's money on the line. Arguing about how "Apple is better because they have more capital resources" has about as much pragmatic value as a child's crayon drawing.

Unless Apple has one-upped Room 641A, I think you're describing a power fantasy.

Right, but I think there's some incorrect conclusions being drawn.

The article asserts that an error in the settings menu appears: "Your cellular plan doesn’t support iCloud Private Relay. With Private Relay turned off, this network can monitor your internet activity, and your IP address is not hidden from known trackers or websites."

This doesn't appear to just be a situation where T-Mobile started blocking it at the network level; it appears to be one where Apple submitted.

While there's a lot of theories in this comment about how Apple will respond; I don't see that happening (in a public way, of course). Apple's leadership in 2022 doesn't have the same convictions their leadership has had in the past. They're capable of being a positive force for change, in fair weather; but when the weather gets rough, or when forces assert power over their expression of values, they fold.

What, so now displaying an error message means you're responsible for the error? See some of the other threads about why Apple might not like playing dirty to go behind T-Mobile's back--each one needs the other for its good reputation.

Its reasonable to assert that they wrote the error, and they phrased the error message intentionally, in a way which clearly says that they expected carriers to block the service. The settings app is owned by Apple; not T-Mobile; T-Mobile would certainly NEVER admit so plainly that they monitor network activity (even though they do).

Alternate phrasing which betrays different expectations: "We could not connect to the iCloud Private Relay servers. This may indicate an issue with your network provider, blah blah blah."

No VPNs is mostly standard-operating-procedure in, say, China. That being said: I'd assume that feature, let alone the settings page to configure it, is hidden in versions of the software distributed in countries like that. This error message is likely for countries where the service is available; just not on your carrier.

But putting that aside and even considering their stance of submission to the CCP; they betray every spoken value their American executives verbalize. That is standard operating procedure for 2022 Apple, and most other gigacorporations. That is the lens that every statement Tim makes, every word spoken at their keynotes, needs to be viewed through; that they're willing to invest their infinite money in whatever projects they believe aligns with their values, but they're wholly unwilling to stand up for those values when those projects are battle-tested in even such an absolutely inconsequential way as this.

Of course, they can prove me wrong by standing up to T-Mobile and using them as an example. I mean my god, you couldn't ask for a better example to make, T-Mobile/Sprint is a fourth-rate bargain bin cellular carrier, we're not talking about a nation state; this is a toddler mad at his parents because they won't let him eat candy for dinner. If they can't even resolve that, what hope do any of their values have?

The wording "doesn't support" seems very cooperative; it's less likely to generate angry calls to the network's customer service than "sabotages", "blocks", or "interferes with" - all of which are arguably more accurate descriptions.

Apple was the one who wrote that text out and put it in your iPhone. You can choose to interpret that any way you choose, but it's pretty clear that Apple either really loves and trusts T-Mobile or (more likely) their "Privacy is a Human Right" bit rides shotgun to their moneymaking shtick.

>the iPhone was AT&T exclusive for a long time at the beginning.

I think that was a very different time though. Smartphones were just becoming popular. A lot of other upcoming smartphones also had carrier exclusives at that time (Verizon with the Droid line). I don't know if that would be acceptable in today's world.

A joint move like that by the carriers would be subject to a lot of antitrust scrutiny, where as apple can move on it's own with a lot less scrutiny.

So you want them act as monopolists which is something HN usually is very much against. Also the threat of trying to buy a carrier out is a completely empty one and the carriers would all know it. There is no way it would pass regulatory or court review for the market leader in cell phone sales to own a major carrier.

I agree with you that they probably wouldn't be allowed to buy a major US carrier. But they aren't a monopolist or the global market leader in cell phone sales. Even in the US where they are indeed the market leader, their percentage of sales hovers around half, well below monopoly levels.

Anti trust starts when abuse of a powerful position begins. There is no threshold requiring you to be a monopolist or even a dominant player. Just that you abuse your position.

I wouldn't be so certain of that. Antitrust review of mergers is mainly concerned with whether the merger will reduce competition in a market. Since Apple doesn't currently operate a mobile network and none of the carriers currently manufacture phones, it would be hard to argue a merger would lessen competition.

Now whether Apple shareholders want Apple operating a mobile network is a completely different question.

if i had to hypothesize why T-Mobile are doing this, its streaming media.

TMobile has numerous pay-for-play access contracts in place for companies like netflix and hulu. in return they get a QoS tier and guaranteed minimums for their subscribers.

conversely, as others have mentioned and the article itself, private relay is absolutely haram. it damages tmobiles ability to deliver edge content from their contractually obligated players like netflix (without a region netflix quality might suffer) and it completely sidesteps all of TMobiles lucrative user plans that include access to streaming media as a feature relative to the users data cap.

increasingly private "anything" on a cellphone is becoming a hostile proposition for carriers as their revenue is largely based on predatory surveillance capitalism. without metrics and metadata, theyre no different than the water company.

Apple could theoretically attempt (or at least threaten) a hostile takeover of any of them

Yeah, that's not Apple's style. Besides, they don't want all of the legacy crap that comes with owning a cell phone carrier.

I suspect the issue is making its way up the corporate ladder through various directors and Vice Presidents.

And I'm sure Apple's got a business plan for running their own MVNO they could dust off if things get out of hand. But let's hope it doesn't come to that.

Theoretically they could buy it, replace the management, sign a contract to never do such things again, and spin it off. They wouldn't necessarily have to deal with it beyond that.

But this would be rewarding existing ownership with a buy-out.

> People would need to buy iPhones from Apple and then replace the SIM cards themselves.

Changing sims is VERY easy, but a sim that doesn't match an approved phone is also easy to block?

While I agree with your comment almost fully, I think it's a bit too early to judge Apple. They probably found out just a few days earlier than we did and are still weighing their options.

Why would Apple need carriers and vice versa? 3GPP is a series of standards for a reason.

This blocking is a net neutrality issue. Hope it doesn't hold, at least in Europe.

There is no chance Apple would be allowed to buy or run a mobile network. The monopoly dogs would be at the door before the email went out.

Or the other cellular networks could start running ads touting that they let iPhone users use all of the iPhone features.

"Does your cell phone company hold you back? With Cincinnati Bell, you can do things with your iPhone that T-Mobile won't let you."

Apple could even help pay for the ads. It's not like companies with aligned interests don't do ad cost-sharing all the time anyway.

> I would think Apple has some leverage to force it if they really wanted.

No, because then people will ask why they also don't use the leverage to enable this feature in China/Russia/...

Apple has to abide by the law in each jurisdiction it operates in or cease operating there.

It doesn’t have the ability to give a middle finger to both those governments and still be able to sell devices to Russians and Chinese people to use Private Relay on.

Last I checked, T-Mobile has approximately 100% fewer nukes than Russia or China.

I don't think anyone thinks America's third-largest cell carrier has the same power as the Russian or Chinese government. Certainly Apple can say "We cannot make China and Russia do what we want" because they don't have that much leverage.

I think you should avoid T-Mobile if you can. Not just as a matter of principle, but also pragmatism. They have an extremely crude SMS censorship/anti-spam system [1] which even blocks links to lichess.org, the popular online chess website.

They have poor security practices like storing passwords in plaintext [2], and they had a large data breach (probably about 100M customers affected) last year. [3]

A̶n̶d̶ ̶n̶o̶w̶,̶ ̶i̶t̶ ̶s̶e̶e̶m̶s̶ ̶t̶h̶e̶y̶ ̶a̶r̶e̶ ̶t̶h̶r̶o̶w̶i̶n̶g̶ ̶i̶n̶ ̶s̶o̶m̶e̶ ̶p̶r̶o̶t̶o̶c̶o̶l̶ ̶b̶l̶o̶c̶k̶i̶n̶g̶ ̶t̶o̶o̶.̶

PS: This isn't protocol blocking at the packet/port level, so I may have used "protocol blocking" a bit inappropriately. Apparently Apple allows the carriers to prevent people from enabling iCloud Private Relay, and T-Mobile is doing that. Apple is probably doing so due to the pressure by the carriers. In August, four carriers (Vodafone, Telefonica, Orange and T-Mobile ) signed a letter urging the European Commission to stop Apple from providing Private Relay. (According to a report by The Telegraph: https://archive.fo/BRUS4#selection-915.74-925.194) This, of course, still quite preposterous.

[1]: https://news.ycombinator.com/item?id=29744347

[2]: https://news.ycombinator.com/item?id=16776347

[3]: https://news.ycombinator.com/item?id=28192423 (The first comment by @jonathanmayer has a list of other recent T-Mobile security incidents)

I would love to leave T-Mobile, but they are the only carrier in the US who offers such a core piece of functionality for me: International service included out-of-the-box.

I love to travel, and nothing beats being able to land in (pretty much) any country in the world, turn on your phone and have working service just like that. No SIM cards, no different numbers, no local pre-paid cards, and no crazy international fees.

As someone who enjoys work/travel for weeks to months at a time, every other major carrier is not feasible for this (think 10$/day, which becomes unreasonable when you're out of the country for 3+ weeks).

Unless somebody else could recommend another option it seems I'm stuck with T-Mobile for now.

This is the only reason I switched to T-Mobile originally and the only reason I still have them. Their coverage is so poor that I get no LTE service sitting in my house in a core part of the major metro area. I'm only able to maintain them because they were an early and ardent adopter of WiFi Calling. On a recent trip in the US I had no service off major interstate highways. Internationally though, T-Mobile is amazing. I honestly wish my experience in the US was as good as my experience while traveling... there's not much point in having uncapped LTE when you get 1 or 0 bars of service, at least internationally I get great service even if it is speed capped at 256kbps.

I now do Unlocked phone that supports eSim + Airalo app for intl data packs + Mint Mobile for the US, this works extremely well and is very cost effective.

With Wifi calling + texting, you can even use your US number internationally since it will work off the eSim data.

Google Fi does everything you ask for (and works with more phones that just those that Google manufactures).

Google Fi is an MVNO of T-mobile/Sprint (last I checked anyways). so if T-mobile blocks the private relay for their network, it could affect them too.

Also, Google Fi kinda sucks. They used to be the cheapest, but nowadays you can get better prices from other services. For example, Google charges $10/gb/mo, whereas Mint Mobile (another T-mobile MVNO) charges 4gb for $15/mo, or $30 for unlimited.

Google Fi is only cheaper if you use less than 1.5gb of data per month, and the service quality is probably the same.

...and that's not even mentioning all the privacy concerns attached to Google.

The difference is that Google Fi runs at the top network priority. You can find loads of dirt-cheap MVNOs, but your data is at the back of the line if there's any congestion.

Getting top priority isn't worth that much of a premium.

It's exciting to watch how many years in a row they can stay at $10/GB, and not in a good way.

As someone who used Google Fi for a while internationally, DO NOT get Google Fi! So many problems on an iPhone 7. Little to no connectivity in many places where they advertised having connectivity. This was ~2018-2020, so maybe it has improved, but I had such a bad experience with them.

I have a newfangled 5G phone that doesn't do 5G even though I'm in Los Angeles, so I was glad to switch from VZW to TMo, to make them buy my phone, then I switched back to VZW because TMo doesn't have 5G either, no matter the maps they show on TV.

I'd do Cincinnati Bell in a heartbeat, if I could.

Oh god, their service was EVEN worse in the United States!! In major metropolitan cities, in San Francisco especially! I have never had a worse experience with a cell provider. And I use AT&T!

I have AT&T and it's a toggle to turn it on but you're right about the $10/day. I've felt the sting many times.

When I travel anywhere for more than 5 days, I typically just buy a local sim card when I land. Usually saves a lot of money.

Sure, people can't call me on my usual number, but I see that as an additional benefit. I'm on vacation.

Google Fi uses T-Mobile in the background. Depending on what you mean by "service probably isnt [sic] as good", you may either be wrong or be making a niche point.

And go where? I've had bad experiences with service with AT&T and Verizon in my area, Washington State. It's shockingly spotty.

avoid T-Mobile and join AT&T or Verizon? i'm sure they have their fair share of shady/borderline illegal things they do.

> i'm sure they have their fair share of shady/borderline illegal things they do.

That might be true, but at least AT&T doesn't block private VPNs, nor has plans to do so.

Good point. I upvoted your comment, and I wish I could edit my comment to add this. It seems that I can't edit the comment anymore though.

PS: The 9to5mac article has an update which indicates that the blocking is _not_ exclusive to users who have "filtering and blocking features enabled".

Dude, t-Mobile is leagues better than AT&T or Verizon.

AT&T essentially bankrolled OANN, Trump's propaganda network. Verizon has done tons of shady stuff in the past as well.

Plus my bill for TMO has been constant (like the same) for years straight - no overages or surprise bills. Not going back to ATT/VZ for a long time if ever.

Unless someone can spare me $70,000 so I can pay the install fee for a Comcast Internet connection to my home in downtown Chicago, I have to stick with T-Mobile for my main Net connection.

Like I pointed out in the sister thread about EU telcos:

https://news.ycombinator.com/item?id=29875805

Phone carriers do not want to be a dumb pipe - and having Private Relay go through their networks breaks:

- HTTP header enrichment (which they use for self-care/customer sites/services),

- zero rating (which they set up deals for with social networks, music streaming services, etc., often applying specific QoS tags) and

- all sorts of value added services (many using deep packet inspection and DNS analytics) that they offer instead of raw, unfettered connectivity.

I don't think many people are aware of exactly how much data telcos are sitting on, anonymized or not.

And, of course, it also plays havoc with legal interception because there is no easy way to do MITM.

(edit: readability)

Is this really about EU telcos, though? In the European article I mostly see messages about this from UK telcos, which are European but no EU anymore. I've heard that UK net neutrality law is kind of a joke, and now that they're outside of EU control the UK can do whatever the hell it wants, and I fear for UK citizens that the mostly consumer-focused EU ideals aren't shared by the current UK leadership.

Plenty of telcos want to force competitors out of the market with zero rating and triple play subscriptions, but I don't think any of them have made any moves against net neutrality this bad. A few years ago I've seen carriers doing HTTP introspection to force images through their compression proxies (usually budget ISPs who want to stop people from actually using up their data plan so they can make a profit) but that seems to have stopped completely now.

As for legal interception, this doesn't make any difference. When law enforcement finds that the suspects are communicating over Apple's network, they'll just knock on Apple's door with a warrant and demand a wire tap from their network. That's how legal interception of "privacy protection" VPN providers works, and Apple isn't even trying to ship traffic outside national borders, just to the closest data center.

> Is this really about EU telcos, though?

From that article:

> Via The Telegraph, operators including Vodafone, Telefonica and T-Mobile signed an open letter voicing their opposition to the rollout of the feature.

Those are each large telcos with operations in a multitude of countries.

But the operations in individual countries are legally separate companies, aren't they? Plus they still have to obey local laws in every country.

Exactly, carriers really don't want anything that helps push net neutrality in any real way. They don't want to be commoditized to where it's just pipe for Internet data to transmit through as you mentioned.

Shouldn't those all be true of ISPs too, though? Why are telcos different? Is it just because they need stricter QoS because of airwaves vs. cables? Do you think that argument still holds water in a post-5G-saturated world?

MITM is pretty moot right now with TLS everywhere. Apple is taking this stand because it’s inline with their business.

Zero-rating is really bad for Apple. And by making themselves the virtual network layer, they have the ability to roll out their own last mile networks later.

To be fair, you could make the same argument that TLS is moot because everything at the other layers (routing, application, and even hardware) is extremely vulnerable to attack. MITM is still a very real thing.

If anything makes it moot, it's not other technology; it's social engineering attacks.

The point of TLS is that every bit of network infrastructure could be compromised but your connection would still be secure as long as your own device and the end server (and the cert authority) remained clean.

One big difference in the US is that most telcos also have ad businesses, and this will negatively impact them.

I doubt long term that it causes much havoc with three letter agencies. If anything, it simplifies it a small bit because now they can look at the records of only two intermediaries, Apple and the CDNs they use. That said, why go to the trouble? Depending on how it's configured, Apple would already likely be tracking your browser history in iCloud, backups, etc. Plus websites that track user activity (e.g. have logins) can be asked directly for data.

I have been working through some consulting activities with 8 telcos over the past years on the topic BiG DaTa. While it is true that telcos have data, ALL of the telcos I have worked with lack the capability to do ANYTHING with that data.

First, they dont get the right people, because good people dont go to telco. Second, they have super fragmented stacks, especially in markets that have consolidated over the years. Third, they simply dont have figures out ANY business model for that data (except some We SeLl LoCaTiOn DaTa To GoVeRnMenTs that is illegal in most Western countries anyway by now).

So... all this "TELCO SOOO BAD BECAUSE ALL MY DATA THEY EAT" talking is laughable to me after seeing the truth. I am surprised what people here in HN think of the capabilities of telcos.

Edit: as I saw some comments below on "three letter agencies". Fun fact, ALL the 8 telcos that I have experienced hat guys from the local "three letter agencies" working there to detect crime stuff.

I just tried it. I'm on the cheapest T-Mobile prepaid plan [1].

I turned off WiFi, turned on iCloud Private Relay, and browsed to a site at work while watching the Apache logs for the hit.

Site showed up, and the Apache logs showed the hit came from 172.224.242.xx, which is in the block 172.224.0.0/12 which is assigned to Akami.

I turned iCloud Private Relay off, hit refresh, and the hit came from 172.58.46.xx, which is in the block 172.32.0.0/11 which is assigned to T-Mobile.

I then turned WiFi on and iCloud Private Relay on, hit refresh, and it came from the same IP that it had with it on when WiFi was off.

I then turned iCloud Private Relay off, hit refresh, and it came from an IP in the block 73.0.0.0/8 which belongs to Comcast, which is my wired home ISP.

Looks like there is no blocking going on for my T-Mobile plan.

[1] https://prepaid.t-mobile.com/plan-detail/t-mobile-connect

>The change does not appear to be network-wide just yet, but rather it appears T-Mobile is in the process of rolling it out.

Can anyone explain the case from T-Mobile's end?

(Not asking for sarcastic not-in-good-faith explanations of BS reasons that you are imagining.

Asking for anyone who understands more about a cell carrier's needs than I do, to explain what «the feature cuts off networks and servers from accessing vital network data and metadata and could impact “operator’s ability to efficiently manage telecommunication networks.» actually means, to someone who is not a telecom engineer but does understand engineering.

And/or other motives, but based on understanding more of their business than I do, not just wild guesses!)

These aren't wild guesses, but I also don't have inside information.

1. Browsing history. We know that Verizon is tracking it for their gain: https://www.wired.com/story/verizon-user-privacy-settings/. It seems reasonable that T-Mobile and others don't want that door to close on them.

2. Video streaming management. Carriers typically restrict video streaming on some/all of their plans to certain resolutions. For example, I think most American carriers limit video streaming to around 480/720p at 1.5Mbps or less unless you have bought a premium plan. VPNs often get around this and I know that my carrier can't detect Netflix access through iCloud Private Relay. Right now, iCloud Private Relay doesn't proxy app traffic, but it could in the future.

3. It looks like mobile carriers are looking to get into "edge cloud" stuff. Verizon has been pushing this and they recently emphasized this in their 5G Ultra presentation. If traffic is going through iCloud Private Relay, buying expensive "edge cloud" services from Verizon is a waste of money since the traffic would be leaving the network to go through Private Relay.

3a. Netflix ships "Open Connect Appliances" that ISPs can hook into their network to serve Netflix content. If your traffic is going through a proxy, you start accessing the content on a server farther away. This mostly doesn't apply given that Private Relay only does Safari traffic, but one could see Private Relay expanding to apps in the future.

4. I think there is a certain knowledge of what is using data that can be helpful to carriers. For example, I worked for a university and they wanted to set different QoS for things like peer-to-peer file sharing vs. web browsing. The university didn't want to punish P2P tech or anything like that. They just wanted to make sure that P2P usage didn't overwhelm other users and uses of the network. Likewise, it could help the university spot patterns like viruses/bots that might be using a lot of network traffic.

4a. I think this can also play into how companies position their offerings. For example, T-Mobile has introduced features like "Music Freedom" and "Binge On" that allowed unlimited audio streaming and video streaming before unlimited plans were a thing. They surely did analysis of network usage of those features before introducing them. You can look at how much video streaming users are doing and then model how much data would be used if you limited it to 480p (including accounting for an uptick in usage due to it being unlimited). However, if you don't know how data is being used, you lose the ability to spot patterns that might be opportunities.

4b. It makes sense to want to offer different QoS for different services. If someone is using FaceTime, you want that to be a good experience. You don't want to prioritize a speed test over someone's FaceTime call. You don't want to prioritize downloading from YouTube over a FaceTime call. That YouTube video can be buffered and if you know that you've transferred 15 megabits worth of 1.5Mbps video, you kinda know that the user doesn't need the next 1.5 megabits of video for 10 seconds.

4c. I know that a lot of people want their connection to be an unbiased dumb-pipe, but I think that people only want that because they tend to see crappy stuff from companies looking for money. Seeing it from a university that only wanted to give people the best possible network experience feels a bit different. QoS can be a positive thing and a dumb-pipe isn't always great.

I'm a bit surprised that T-Mobile would go this route at this time. iCloud Private Relay doesn't proxy app traffic at this time and I haven't seen that they have a similar browsing-history program like Verizon's. Still, there are reasons to want to be able to understand your traffic both for business reasons and for a better customer experience. Again, I'm surprised because it seems like the reasons today are slimmer. I think the Netflix OCA use case is a good one since it reduces network usage in a way that simply helps the parties involved, but wouldn't really be possible if the traffic first went via another external server.

I'd emphasize that nothing here is to say that T-Mobile is doing the right thing. It's just to bring up areas where a company might want to know more about its network access patterns. Some of that can be used for good like the Netflix OCA system or giving higher QoS guarantees to FaceTime. Some of it can be used for bad like knowing using browsing history for advertising.

Very good point on cache servers for Netflix and YouTube. Telcos save a lot of money when they terminate the traffic in their network. Private relay may affect that. Although apple’s solution could be clever about it.

The reason is right in the "what's new" section of the T-Mobile privacy policy: https://www.t-mobile.com/privacy-center/our-practices/privac...

> "However, starting April 26, 2021, T‑Mobile will begin using some data we have about you, including information we learn from your web and device usage data (like the apps installed on your device) and interactions with our products and services, for our own and 3rd party advertising, unless you tell us not to."

T-Mobile sells browser history data to advertisers, and Private Relay blocks that revenue stream. They are on the offensive to protect their new-found profit center, and most likely are doing this now to show Apple that this is not a feature that they want to see be turned on by default.

It's the beginning of the same saber rattling that Facebook did when Apple announced it would simply ask customers if they wanted to allow apps to track them

On cell networks, video content is by far the largest consumer of bandwidth. And the default for video generally is to auto-adjust the resolution to the highest quality that the network supports. This kind of sucks, since bandwidth is a shared resource for all users of a given antenna on a cell tower.

Though Speedtest on your cell might show your connection speed as 100 megabits/sec down, cell networks special-case video by identifying it as video and rate-limiting it to something like 1 megabit/sec. This is considered "efficient network management". For T-Mobile, this based on the plan (https://www.t-mobile.com/cell-phone-plans), they sell either "SD streaming" or "4k UHD streaming". "SD streaming" is a fancy way to express that they rate-limit identified video streams to 1 megabit/sec.

They identify video streams by watching the IP your phone is connecting to and/or the hostname mentioned in the TLS SNI header and checking if it is Youtube, Netflix, etc. Sending video content over a VPN removes their ability to understand what the content is.

There is a solid, technical problem with VPN usage on such a massive scale. Carriers, like T-Mobile, can arrange traffic exchange with big content providers. Majority of traffic generated goes to a handful of providers, like YouTube, Netflix, Facebook. It's not even about direct, financial incentives. It's a win-win for both ISP and content providers to peer directly and limit the amount of traffic routed through paid uplinks. It's a win for users too, since they can get their content with less hops, through bigger pipes. Even Tier-1 network operators (https://en.wikipedia.org/wiki/Tier_1_network) can optimize traffic by making the direct inter-connections for traffic-heavy content.

When everything is encrypted and goes over the ISP just to the VPN endpoints, they can't do anything. In the end, they will have to arrange peering not with content providers but with VPN providers, who works for Apple.

PS. There is a lot of tension in current setup, even without Apple stepping up. In the old fashion market, the last mile is the king. Big grocery chains have direct access to users, so they are the strong side in the relation with producers. They can position brand X over Y, if they have better margin. They also create their own brand Z rip-off and sell that directly. Just look what Amazon does in that space. When it comes to ISP, they have direct users and have very little to say. They are basically dump pipes, just like the power line.

T-Mobile was very vocal in the past in that space. They often wanted the MANGAs (heh) of the world to pay them a share from their ads. I remember T-Mobile threatening, that they might replace some ads with their own ads. Since they provide the users with phones, they can install their own certs on devices. Chrome has SSL pinning not only, to save users from hackers, but to save their own business model being attacked by ISPs.

Wouldn't most carriers have peering relationships with Apple for Apple TV content? Presumably iCloud Private Relay traffic could be sent over that connection as well, meaning that of all the VPN-esque solutions, Apple's is the most efficient from a traffic exchange perspective.

(Of course, it was likely never about the traffic exchange itself, but rather the ability to route, shape, and track traffic dynamically on a host-by-host basis, as others have speculated.)

I am guessing that AppleTV traffic is very low compared to Netflix or YouTube. Anyway I guess this setup must be super complicated. Apple is probably partnering with tons of entities to provide this all over the world. It must be a huge challenge at Apple scale. I wonder if they have any long term goals, which are not obvious.

Well the ISPs and content providers can cry me a river about their network optimization. They brought this all upon themselves. Remember when nobody used to use VPNs?

I belive this functions like a VPN in some ways and blocks video throttling. They use traffic inspection to throttle video streams down to 480p unless you have the most premium of plans.

I've never heard that IPS (not the content provider) is throttling down Video Quality by altering the traffic. Do you have some links to back up that claim? This doesn't make much sense, as they would have to download the high quality video anyway, then invest massive CPU power to downscale this. Most content providers will scale down the quality if they detect bad network conditions. If ISP would want lower quality, they could just artifficaly slow the connection.

This has been happening for years. They just limit bandwidth to video sites, so your player selects the corresponding stream quality.

Non-cynically, it probably does introduce some issues in these legacy telecom systems.

For example, if you run out of data for a month, many carriers will continue giving you access to the internet APN, but then block access to "external" websites. This is so you can easily open your browser and "top up" on data to continue using your device.

Or the usage of HTTP (not HTTPS) was relatively common back when I was in the space (7-10 years ago). There wasn't a need to use HTTP because the carrier was in full control of the pipe between the device and the server. Adding in a VPN that somehow tries to intercept that traffic (that was supposed to exist entirely within the telecom) is not going to work.

But if that were the only reason why, then couldn't they just turn off Private Relay in that specific case, instead of all the time?

No, they're (very likely) using the same mobile entitlements system that does tethering/mobile hotspot and WiFi calling enablement checks (the Entitlement Layer Protocol).

The message says that the user's "cellular plan doesn’t support iCloud Private Relay," so is this the same thing they've done with other VPN providers? That is, do they just count the traffic against the tethering/hotspot limit, since they can't shape traffic on it to, e.g., limit video quality to 480p when a user has a plan with that limitation? I don't know if they actually do this, but I've heard it before.

https://www.reddit.com/r/tmobile/comments/9ja8y1/i_can_confi...

No, they do not allow users to enable Private Relay at all because Apple allows carriers to determine whether it's available or not. Even FaceTime over cellular is still something that carriers get to decide whether to allow or not, although I'm not aware of any carriers that don't.

From my limited testing, carriers are whitelisting traffic for high-bandwidth. When I establish a vpn tunnel on my Tmobile sim card, bandwidth drops dramatically. Presumably because they can't inspect it.

There are legitimate reasons why a specific business network might not allow it. For example, if you're on the employee network of a bank or hospital, it's very likely that your web connections are going through a proxy to make sure you're not sharing confidential data, and to block malware and such. Private Relay would go around those proxies. Allowing networks to opt out of Private Relay, then, is a better business decision than having enterprise networks just block all iPhones.

Corporate networks makes sense, but giving carriers the ability to disable it on the phone (i.e. not via blocking mask.icloud.com) doesn’t make sense. It’s not like personal hotspot where it allows you to bypass network policies, except for maybe the streaming shaping (but how long did they think that would work anyway?).

Because Apple wants to keep their carrier partners happy, so they give them control over things that will have an impact on cellular data.

Like I noted with FaceTime over cellular, it's nothing new.

I can't imagine what kind of leverage they think they have. is any provider going to just drop iPhone support from their network?

These deals are old. FaceTime when it came out was in the era of 3G. FaceTime over 3G could be a bandwidth hog.. and iPhones were not nearly as popular, so the negotiations were more give-and-take.

If I had to speculate, in order to continue operating in regions where governments more tightly control carriers.

This is the worst thing. Not for Apple or Apple users, but for the general internet. If that goes through, and countries effectively end up making Private Relay illegal, that is a very VERY strong precedent to block regular VPNs. And that's terrible.

I wonder if the same could happen to TOR, if VPN end up the same way...

No, it is because...

> The carriers wrote that the feature cuts off networks and servers from accessing “vital network data and metadata and could impact “operator’s ability to efficiently manage telecommunication networks.”

But seriously, it is because it prevents T-Mobile from monetizing you and slowing you down.

It also cuts down on the number of companies they can extort for transit. Right now they can go to Netflix and say "would be a shame if T-Mobile customers couldn't view movies during peak hours" and Netflix has to pay them for that not to happen. With all the traffic going through Apple, Apple is the only company they can extort this way. (Meanwhile, Apple or their "third-party provider" could of course play this game, but historically tech companies have been super uninterested in doing this.)

Basically, what everyone wants is for companies like T-Mobile to be a dumb pipe. They invested in spectrum and a network, and they should just lease that network for cost + profit margin. Instead, they want to milk it. They want you to pay more for particular packets. They want the rest of the Internet to pay more for particular packets. They want to inject their own ads into unaffiliated websites. They want to build a marketing profile based on what sites you visit, and send you "offers" based on this. Right now, that is all technically possible, so they'd be defrauding their shareholders if they didn't try. But, we can of course say "no" and route around the damage. Apple is letting their customers say "no", and that means T-Mobile is doomed to irrelevance, and that's a great thing. Infrastructure should be infrastructure.

(Can you imagine what it would be like if other utilities did this kind of shit? Your water would cost less if you were using it to run a Coke-branded soft drink dispenser, but not a Pepsi one. Or, Dell computers could get electricity at a 10% discount, but not Asus ones. It would be unthinkable! But with these big ISPs, it's mandatory.)

I hate to reply to myself, but I wanted to say one other thing. When governments sell RF spectrum to companies, the expectation is that they become good stewards of the shared resource. The taxpayers are saying "you know, we think private industry can give us more value from our RF spectrum than the government", and this is their chance to prove that. What we didn't want was to enable a monopolist to nickel-and-dime the Internet to death.

I'm guessing the exact legal agreements didn't spell it out like this, but that's how I think of it. Only one company can use this finite resource at once, but just because they bought it doesn't mean there is no limit to what they can do with it.

> Right now, that is all technically possible, so they'd be defrauding their shareholders if they didn't try.

This sounds like a clumsy restatement of the urban legend that companies have an obligation to maximize shareholder value. There is in fact no such rule, for the obvious reason that nobody can accurately predict the future and calculate the optimal value.

https://corpgov.law.harvard.edu/2012/06/26/the-shareholder-v...

In this case, a company like Apple could say that they are choosing to forgo short-term profits from selling out their users’ privacy because they feel that the long-term loyalty will be greater, and anyone arguing otherwise would still have to admit that this approach has been phenomenally profitable.

Does T-Mobile actually extort companies for transit? When they announced their video streaming throttling + zero-rating, I looked through their the publicly available documents. From what I recall, there wasn't any sort of payment process, and mostly there was two parts: identifying the traffic so T-Mobile knew to zero rate it, and either adaptive bandwidth usage (which seems pretty common for video streaming anyway) or identifying the traffic so the provider could serve lower bandwidth streams.

It's not in line with the net neutrality, but it's useful for the direct parties:

a) a video streaming customer wins because they can do video streaming without touching their data allotment.

b) the video streaming server wins because their customers are able to do more streaming

c) t-mobile wins because they've reduced bandwidth requirements

Competitive streaming services that are not included in the program don't win, but t-mobile made it fairly easy to join. Users who want to stream at 4k or whatever don't win, but they can turn off the bandwidth restrictions and use their data allotment if that's what they want to do.

At my last job, I was involved with a lot of zero-rating deals as the application provider; we never paid for it, and I don't recall ever being asked for payment. Some of the carriers even setup plans without our knowledge or consent or assistance; this didn't usually work great long term, because of misidentified traffic, but it indicates the demand was there without us pushing it.

> Right now, that is all technically possible, so they'd be defrauding their shareholders if they didn't try.

Can you expand on this? Are you saying that if a business opportunity exists and a company elects not to pursue it that constitutes defrauding shareholders? I would have thought it constituted nothing more than a disagreement over strategy.

It sounds like a sarcastic statement of the "profits not gained is profit lost" mindset and that shareholders would be upset, not literally a crime.

Yup, you're exactly right. I wrote it without thinking much (but wasn't fully serious), and didn't realize it would generate this much discussion. That's my bad for being a little careless with my rant. Thanks for clarifying it a little on my behalf.

> Meanwhile, Apple or their "third-party provider" could of course play this game, but historically tech companies have been super uninterested in doing this.

Apple notoriously "extorts" developers to be in the app store.

> Basically, what everyone wants is for companies like T-Mobile to be a dumb pipe. They invested in spectrum and a network, and they should just lease that network for cost + profit margin.

I don't think you've considered the alternatives if T-Mobile can no longer monetize traffic:

* Go back to subscribers pay per kb usage

* Eat the costs themselves

* Raise cost of mobile data plans

> Can you imagine what it would be like if other utilities did this kind of shit?

They side step this problem by charging per-use. During peak demand, prices go up. Each customer pays their share. Downside see Texas snowstorm.

> Go back to subscribers pay per kb usage

They charge $70/month for “unlimited” data which is only 50GB before throttling. I’m pretty sure they can profitably afford to run a network for that much without reselling user data.

They already charge per kb. Look at the small print -- once you hit a certain amount of usage, you are drastically rate-limited. The only difference is that some months, when you don't hit your limit, you pay more per byte.

I really don't see the horror that would be carriers charging for usage. I would rather that than pay for stupid things like "lines" or "devices."

iCloud Private Relay isn’t like full blown VPN that hides everything you do on the internet, only your web browsing in Safari goes through it. So their existing systems to throttle the connection of your video streaming apps will continue to work just fine.

It’s completely about monetizing your browsing history.

I believe it also takes non-https traffic from apps but since they made https mandatory quite some time ago now I suspect that is not much. Also content loaded inside email in Mail.

IIRC it redirects DNS queries system-wide as well which definitely would hinder general interest tracking.

Thank goodness the carriers can't do anything about solutions that use VPN to override default nameservers

Tmobile deprioritizes devices depending on high usage. Private Relay would allow individuals who are deprioritized to bring down entire cell towers.

No, it wouldn't. They'd still have the ability to throttle individual phones generating lots of traffic.

Carriers nat/proxy everything and in addition to bandwidth throttling, they will rate limit or otherwise whack misbehaving applications.

VPNing everything at scale will impact that monitoring/management. And that will absolutely impact towers, or cause the carriers to throttle users vs apps.

...they throttle at the phone-number/SIM. Even with a VPN your phone is still auth'ing itself to the cell towers, and those towers know what device is sending which traffic.

What this prevents is allowing say Youtube to pay TMobile to never throttle their traffic.

VPNs work at a higher level. They have to see the radio traffic to be able to deliver packets to your phone, which is where billing and access control happens (this is why you can’t spoof someone else’s IP to avoid paying your bill), and at the IP level your VPN traffic is carried from your carrier-issued IP address to your VPN provider’s addresses.

The one legitimate argument here is that this prevents traffic shaping based on the destination, which T-Mobile uses to do things like offer unlimited streaming separate from your general data quota.

Currently:

P ---- CT ---- S

With VPN/whatever:

P ---- CT ---- VE ---- S

P = Phone

CT = Cell Tower

S = Server

VE = VPN endpoint

So given this the cell tower can still determine who is using lots of traffic, they just can snoop on that traffic.

You're a little off, currently: P --- CT --- NAT Proxy/Traffic Shaper --- Possible MITM host --- S

Private Relay only touches traffic from Safari, and while people could watch Netflix in the browser instead of the Netflix app, I doubt that many do

I've always wondered if you could start a internet speed testing website, get in the trusted list of companies like T-Mobile. Then release a VPN on the exact same servers, forcing the companies to provide the best speed to the VPN.

Only problem is that you would have to be large enough that the ISPs would care if their scores looked bad.

This is basically what Netflix did. They launched fast.com, which comes off the same servers as Netflix video. The whole goal was to get people to call their ISP and complain they aren't getting the speeds they paid for and getting them to unthrottle Netflix.

T-Mobile partners with various video providers to provide lower-bandwidth streams that don't count against bandwidth caps. Less-cynically, this may be to enforce that.

I consider those agreements to be violations of Net Neutrality, since they're inherently not treating all data the same.

It is a blatant violation of net neutrality, but somewhat paradoxically, actually benefits the consumer in my experience. Several friends of mine on T-Mobile have raved about how Netflix/Spotify/et al. don’t count towards their monthly data limit.

That said, iCloud private relay only applies to Safari, so T-Mobile blocking it probably doesn’t have much to do with their variable data caps.

T-Mobile is pretty up-front about the various video quality options with their plans, and also has ways to temporarily boost your video quality for a few dollars.

For many people, a cheaper plan with slightly lower quality video is a great tradeoff.

It's not paradoxical at all, net neutrality also protects from bad effects kicking in long-term. Zero-rating is effectively the same as providing dumping prices compared to the competition. It may benefit the customer now, but leads to lock-in.

See Facebook's internet.org.

> I consider those agreements to be violations of Net Neutrality, since they're inherently not treating all data the same.

I would agree if they do not make that available to all services. At least at the time they did that for music there was a pretty long list of partners so I’d be most interested in knowing whether they charge money or reject applicants.

T-Mobile probably isn't extracting too much of value from HTTPS traffic. It's probably more about traffic shaping.

You can extract a whole lot of value by mapping which sites someone is visiting even if you don't know what they're doing there, and you can get that information just from IPs.

The hostname of most (all?) TLS connections is sent plaintext at the start of a new connection. This is called SNI (Server Name Indication).

That provides some (or a lot) of value I am guessing.

Even without that, it's a pretty easy traffic analysis for:

- Time T0: User requests the DNS record for example.com

- Time T0+10ms: DNS returns "example.com. 193 IN A 10.1.2.3"

- Time T0+20ms: User opens a connection to 10.1.2.3 port 443

Chances are pretty good they're looking at example.com, even if you can't examine a single packet.

Still hides HTTP level metadata like the path, POST body, cookies, etc, no? All you’d have is the hostname

> “vital network data and metadata and could impact “operator’s ability to efficiently manage telecommunication networks.”

Complete bunk - Their (TMobile et.al) “value add services” are nowt more than network content provider toll-gates that the proxies bypass. Meanwhile they are also selling every bit of user context data (position, DNS/sites, cookies where unencryptable, phone-id’s etc) that they can scrape individually and in aggregate to any and every advertiser. Context is worth serious money to advertisers.

If carriers could be trusted (and they clearly can't), I'd actually agree with some of their technical requirements. Netflix's edge boxes work well to keep them from wasting peering capacity on video streams, and dedicated Youtube and Twitch uplinks would save the general-purpose peering links from a lot of unnecessary load. Unmasked routing would help ISPs route their traffic more efficiently and cheaper. Latencies would be lower, and rush hour throughput speeds could be higher. It might even be a small win for the environment to send all of your traffic back and forth between data centers.

Sadly, many (American) ISPs are abusing their position to gather and sell personal information from their subscribers. They wasted their "ability to efficiently manage telecommunication networks" the moment they started selling data. They've become adversaries rather than partners because they thought they could have their cake and eat it too. It's sad, really, because with cooperation, everyone would actually be better off with proper network management!

These kinds of shenanigans are exactly the reason you shouldn't trust carriers with plain text data. People bash Apple for not adopting RCS over iMessage but it would just lead to more crap like this but for your text messages.

RCS supports E2E encryption, and Google's apps implement it.

(And I think the complaints about iMessage are its exclusivity - the best solution is an iMessage for Android.)

Thanks, I missed they'd added E2E last summer. It looks like it's only for 1:1 chats and only on some phones depending on handset vendor and carrier, is that accurate? If so it still seems like adding RCS would have pretty limited usefulness vs interop with say WhatsApp.

I don't fault any one company on the messiness of the situation, it's kind of a tragedy of the commons situation. Apple isn't willing to compromise the UX complexity of adding more messaging types with different behavior, Google isn't willing to force carriers and handset manufacturers to make RCS really good, and carriers just don't care about anything other than ARPU and being "value added".

Oh, and WhatsApp interop will never happen even though that would probably actually be good because Facebook.

RCS is a shitty system set up by a shitty telco industry. The protocol is behind what most countries in the world use already. I see it as just an attempt from the telco industry to start charging subscriptions for Whatsapp again, but about five to ten years too late.

iMessage would be fine if it wasn't for the shitty vendor lock-out. Everyone I know uses some kind of cross platform chat app, usually either Whatsapp or Telegram. It's sad to see the green bubble shaming that Apple's exclusionary tactics has created be of such influence in US social circles.

Carriers generally don't care about payloads, they can monetize you from the metadata. What kind of websites you frequent and when. They don't need to know which color of maternity clothes you're shopping for to know you're pregnant.

I previously served as CTO of the FCC Enforcement Bureau. A couple thoughts on the regulatory dimensions of this report.

* This could be a Federal Trade Commission problem. T-Mobile, like all major ISPs, has made public representations about upholding net neutrality principles [1]. These voluntary commitments were part of the Trump-era FCC's rationale for repealing net neutrality rules. Breaching the commitments could constitute a deceptive business practice under Section 5 of the Federal Trade Commission Act.

* This could also be a Federal Communications Commission problem. When repealing the Obama-era net neutrality rules, the Trump-era FCC left in place a set of transparency requirements [2]. Making an inaccurate statement about network management practices can be actionable under that remaining component of the FCC's net neutrality rules.

I haven't seen a comment from T-Mobile, so to be clear, that's just based on the report.

[1] https://www.t-mobile.com/responsibility/consumer-info/polici...

[2] https://www.ecfr.gov/current/title-47/chapter-I/subchapter-A...

> Making an inaccurate statement about network management practices can be actionable under that remaining component of the FCC's net neutrality rules.

Who would be responsible for bringing about that action and, if they don't bring about action, what can regular people do about it?

Does this mean Verizon and t-mobile are also blocking all VPN traffic?

Also, how can the "land of the free" not have net-neutrality laws?

We did, briefly, under Obama. More recently, the previous administration unwound those rules.

More technically: NN was implemented via the existing authority of the FCC, rather than any new law. Then the FCC, under new leadership, decided that internet service was outside of that authority, actually, and dropped that enforcement. Under Biden, there has been no change back in the other direction. (And at no point has there been a separate, federal law.)

The previous administration even attempted to prevent states from having net neutrality by claiming that disclaiming FCC authority was a prohibition on it. Yes, by attempting to claim FCC had no authority to regulate they also simultaneously claimed this prohibited states from regulating it.

The paradoxical was a direct reflection of the corruption within the FCC at the hands of the previous administration.

If anyone is aware of any grassroots efforts to reinstate NN, please comment. I had basically forgotten about the rollback under Ajit Pai, which, is in my cynical view, exactly what they want.

You could say our ISPs are free to make deals with whomever wrt bandwidth.

Is free, unlimited HD Netflix steaming worth more than private relay? I’m guessing most people would say yes.

I’d consider switching. Oddly enough though I was able to turn on private relay on T-Mobile USA.

No, Verizon is not at least. I will commonly connect to my home network over self-hosted vpn while on Verizon LTE.

Same here, except instead of commonly, always, my cellular data won't work if I'm not connected. If they interfered with that I would switch carriers in a heartbeat.

Does this run counter to current net neutrality regulations? Or is this unrelated.

Are there other legal remedies for either the subscriber or from Apple to the ISPs?

As much as I don’t like this, I would suspect it’s fine.

Disabling this feature is a built-in ability of iOS. It doesn’t depend on ISPs treating the traffic differently.

T-Mobile begins blocking iPhone users from enabling iCloud Private Relay in US

news.ycombinator.com | 12 days ago

GH Archive

Open-source developers all over the world are working on millions of projects: writing code & documentation, fixing & submitting bugs, and so forth. GH Archive is a project to record the public GitHub timeline, archive it, and make it easily accessible for further analysis.

GitHub provides 20+ event types, which range from new commits and fork events, to opening new tickets, commenting, and adding members to a project. These events are aggregated into hourly archives, which you can access with any HTTP client:

Query	Command
Activity for 1/1/2015 @ 3PM UTC	`wget https://data.gharchive.org/2015-01-01-15.json.gz`
Activity for 1/1/2015	`wget https://data.gharchive.org/2015-01-01-{0..23}.json.gz`
Activity for all of January 2015	`wget https://data.gharchive.org/2015-01-{01..31}-{0..23}.json.gz`

Each archive contains JSON encoded events as reported by the GitHub API. You can download the raw data and apply own processing to it - e.g. write a custom aggregation script, import it into a database, and so on! An example Ruby script to download and iterate over a single archive:

	require 'open-uri'
	require 'zlib'
	require 'yajl'

	gz = open('http://data.gharchive.org/2015-01-01-12.json.gz')
	js = Zlib::GzipReader.new(gz).read

	Yajl::Parser.parse(js) do \|event\|
	print event
	end

Activity archives are available starting 2/12/2011.
Activity archives for dates between 2/12/2011-12/31/2014 was recorded from the (now deprecated) Timeline API.
Activity archives for dates starting 1/1/2015 is recorded from the Events API.

For the curious, check out The Changelog episode #144 for an in-depth interview about the history of GH Archive, integration with BigQuery, where the project is heading, and more.

Analyzing event data with BigQuery

The entire GH Archive is also available as a public dataset on Google BigQuery: the dataset is automatically updated every hour and enables you to run arbitrary SQL-like queries over the entire dataset in seconds. To get started:

If you don't already have a Google project...

Open public dataset: https://console.cloud.google.com/bigquery?project=githubarchive&page=project
Execute your first query...

	/* count of issues opened, closed, and reopened on 2019/01/01 */
	SELECT event as issue_status, COUNT(*) as cnt FROM (
	SELECT type, repo.name, actor.login,
	JSON_EXTRACT(payload, '$.action') as event,
	FROM `githubarchive.day.20190101`
	WHERE type = 'IssuesEvent'
	)
	GROUP by issue_status;

For convenience, note that there are multiple tables that you can use for your analysis:

year dataset: 2011, 2012, 2013, 2014, and 2015 tables contain all activities for each respective year.
month dataset: contains activity for each respective month - e.g. 201501.
day dataset: contains activity for each day - e.g. 20150101.

The schema of above datasets contains distinct columns for common activity fields (see same response format), a "payload" string field which contains the JSON encoded activity description, and "other" string field containing all other fields.

The content of the "payload" field is different for each event type and may be updated by GitHub at any point, hence it is kept as a serialized JSON string value in BigQuery. Use the provided JSON functions (e.g. see query example above with JSON_EXTRACT()) to extract and access data in this field.
The content of the "other" field is a JSON string which contains all other data provided but GitHub that does not match the predefined BigQuery schema - e.g. if GitHub adds a new field, it will show up in "other" until and unless the schema is extended to support it.

Note that you get 1 TB of data processed per month free of charge. In order to make best use of it, you can restrict your queries to relevant time ranges to minimize the amount of scanned data. To scan multiple tables at once, you can use table wildcards:

	/* count number of pushes between Jan 1 and Jan 5 */
	SELECT
	COUNT(*)
	FROM `githubarchive.day.2015*`
	WHERE
	type = 'PushEvent'
	AND (_TABLE_SUFFIX BETWEEN '0101' AND '0105')

	/* count number of watches between Jan~Oct 2014 */
	SELECT COUNT(*)
	FROM `githubarchive.month.2014*`
	WHERE
	type = 'WatchEvent'
	AND (_TABLE_SUFFIX BETWEEN '01' AND '10')
	*/

	/* count number of forks in 2012~2014 */
	SELECT COUNT(*)
	FROM `githubarchive.year.20*`
	WHERE
	type = 'ForkEvent'
	AND (_TABLE_SUFFIX BETWEEN '12' AND '14')

Daily reports

Changelog Nightly is the new and improved version of the daily email reports powered by the GH Archive data. These reports ship each day at 10pm CT and unearth the hottest new repos on GitHub. Alternatively, if you want something curated and less frequent, subscribe to Changelog Weekly.

GH Archive

gharchive.org | 13 days ago |

14 comments

Neat Air

neatair.app | 15 days ago

Moxie Marlinspike >> Blog >> My first impressions of web3

Despite considering myself a cryptographer, I have not found myself particularly drawn to “crypto.” I don’t think I’ve ever actually said the words “get off my lawn,” but I’m much more likely to click on Pepperidge Farm Remembers flavored memes about how “crypto” used to mean “cryptography” than I am the latest NFT drop.

Also – cards on the table here – I don’t share the same generational excitement for moving all aspects of life into an instrumented economy.

Even strictly on the technological level, though, I haven’t yet managed to become a believer. So given all of the recent attention into what is now being called web3, I decided to explore some of what has been happening in that space more thoroughly to see what I may be missing.

How I think about 1 and 2

web3 is a somewhat ambiguous term, which makes it difficult to rigorously evaluate what the ambitions for web3 should be, but the general thesis seems to be that web1 was decentralized, web2 centralized everything into platforms, and that web3 will decentralize everything again. web3 should give us the richness of web2, but decentralized.

It’s probably good to have some clarity on why centralized platforms emerged to begin with, and in my mind the explanation is pretty simple:

People don’t want to run their own servers, and never will. The premise for web1 was that everyone on the internet would be both a publisher and consumer of content as well as a publisher and consumer of infrastructure.
We’d all have our own web server with our own web site, our own mail server for our own email, our own finger server for our own status messages, our own chargen server for our own character generation. However – and I don’t think this can be emphasized enough – that is not what people want. People do not want to run their own servers.
Even nerds do not want to run their own servers at this point. Even organizations building software full time do not want to run their own servers at this point. If there’s one thing I hope we’ve learned about the world, it’s that people do not want to run their own servers. The companies that emerged offering to do that for you instead were successful, and the companies that iterated on new functionality based on what is possible with those networks were even more successful.
A protocol moves much more slowly than a platform. After 30+ years, email is still unencrypted; meanwhile WhatsApp went from unencrypted to full e2ee in a year. People are still trying to standardize sharing a video reliably over IRC; meanwhile, Slack lets you create custom reaction emoji based on your face.
This isn’t a funding issue. If something is truly decentralized, it becomes very difficult to change, and often remains stuck in time. That is a problem for technology, because the rest of the ecosystem is moving very quickly, and if you don’t keep up you will fail. There are entire parallel industries focused on defining and improving methodologies like Agile to try to figure out how to organize enormous groups of people so that they can move as quickly as possible because it is so critical.
When the technology itself is more conducive to stasis than movement, that’s a problem. A sure recipe for success has been to take a 90’s protocol that was stuck in time, centralize it, and iterate quickly.

But web3 intends to be different, so let’s take a look. In order to get a quick feeling for the space and a better understanding for what the future may hold, I decided to build a couple of dApps and create an NFT.

Making some distributed apps

To get a feeling for the web3 world, I made a dApp called Autonomous Art that lets anyone mint a token for an NFT by making a visual contribution to it. The cost of making a visual contribution increases over time, and the funds a contributor pays to mint are distributed to all previous artists (visualizing this financial structure would resemble something similar to a pyramid shape). At the time of this writing, over $38k USD has gone into creating this collective art piece.

I also made a dApp called First Derivative that allows you to create, discover, and exchange NFT derivatives which track an underlying NFT, similar to financial derivatives which track an underlying asset 😉.

Both gave me a feeling for how the space works. To be clear, there is nothing particularly “distributed” about the apps themselves: they’re just normal react websites. The “distributedness” refers to where the state and the logic/permissions for updating the state lives: on the blockchain instead of in a “centralized” database.

One thing that has always felt strange to me about the cryptocurrency world is the lack of attention to the client/server interface. When people talk about blockchains, they talk about distributed trust, leaderless consensus, and all the mechanics of how that works, but often gloss over the reality that clients ultimately can’t participate in those mechanics. All the network diagrams are of servers, the trust model is between servers, everything is about servers. Blockchains are designed to be a network of peers, but not designed such that it’s really possible for your mobile device or your browser to be one of those peers.

With the shift to mobile, we now live firmly in a world of clients and servers – with the former completely unable to act as the latter – and those questions seem more important to me than ever. Meanwhile, ethereum actually refers to servers as “clients,” so there’s not even a word for an actual untrusted client/server interface that will have to exist somewhere, and no acknowledgement that if successful there will ultimately be billions (!) more clients than servers.

For example, whether it’s running on mobile or the web, a dApp like Autonomous Art or First Derivative needs to interact with the blockchain somehow – in order to modify or render state (the collectively produced work of art, the edit history for it, the NFT derivatives, etc). That’s not really possible to do from the client, though, since the blockchain can’t live on your mobile device (or in your desktop browser realistically). So the only alternative is to interact with the blockchain via a node that’s running remotely on a server somewhere.

A server! But, as we know, people don’t want to run their own servers. As it happens, companies have emerged that sell API access to an ethereum node they run as a service, along with providing analytics, enhanced APIs they’ve built on top of the default ethereum APIs, and access to historical transactions. Which sounds… familiar. At this point, there are basically two companies. Almost all dApps use either Infura or Alchemy in order to interact with the blockchain. In fact, even when you connect a wallet like MetaMask to a dApp, and the dApp interacts with the blockchain via your wallet, MetaMask is just making calls to Infura!

These client APIs are not using anything to verify blockchain state or the authenticity of responses. The results aren’t even signed. An app like Autonomous Art says “hey what’s the output of this view function on this smart contract,” Alchemy or Infura responds with a JSON blob that says “this is the output,” and the app renders it.

This was surprising to me. So much work, energy, and time has gone into creating a trustless distributed consensus mechanism, but virtually all clients that wish to access it do so by simply trusting the outputs from these two companies without any further verification. It also doesn’t seem like the best privacy situation. Imagine if every time you interacted with a website in Chrome, your request first went to Google before being routed to the destination and back. That’s the situation with ethereum today. All write traffic is obviously already public on the blockchain, but these companies also have visibility into almost all read requests from almost all users in almost all dApps.

Partisans of the blockchain might say that it’s okay if these types of centralized platforms emerge, because the state itself is available on the blockchain, so if these platforms misbehave clients can simply move elsewhere. However, I would suggest that this is a very simplistic view of the dynamics that make platforms what they are.

Let me give you an example.

Making an NFT

I also wanted to create a more traditional NFT. Most people think of images and digital art when they think of NFTs, but NFTs generally do not store that data on-chain. For most NFTs of most images, that would be much too expensive.

Instead of storing the data on-chain, NFTs instead contain a URL that points to the data. What surprised me about the standards was that there’s no hash commitment for the data located at the URL. Looking at many of the NFTs on popular marketplaces being sold for tens, hundreds, or millions of dollars, that URL often just points to some VPS running Apache somewhere. Anyone with access to that machine, anyone who buys that domain name in the future, or anyone who compromises that machine can change the image, title, description, etc for the NFT to whatever they’d like at any time (regardless of whether or not they “own” the token). There’s nothing in the NFT spec that tells you what the image “should” be, or even allows you to confirm whether something is the “correct” image.

So as an experiment, I made an NFT that changes based on who is looking at it, since the web server that serves the image can choose to serve different images based on the IP or User Agent of the requester. For example, it looked one way on OpenSea, another way on Rarible, but when you buy it and view it from your crypto wallet, it will always display as a large 💩 emoji. What you bid on isn’t what you get. There’s nothing unusual about this NFT, it’s how the NFT specifications are built. Many of the highest priced NFTs could turn into 💩 emoji at any time; I just made it explicit.

After a few days, without warning or explanation, the NFT I made was removed from OpenSea (an NFT marketplace):

The takedown suggests that I violated some Term Of Service, but after reading the terms, I don’t see any that prohibit an NFT which changes based on where it is being looked at from, and I was openly describing it that way.

What I found most interesting, though, is that after OpenSea removed my NFT, it also no longer appeared in any crypto wallet on my device. This is web3, though, how is that possible?

A crypto wallet like MetaMask, Rainbow, etc is “non-custodial” (the keys are kept client side), but it has the same problem as my dApps above: a wallet has to run on a mobile device or in your browser. Meanwhile, ethereum and other blockchains have been designed with the idea that it’s a network of peers, but not designed such that it’s really possible for your mobile device or your browser to be one of those peers.

A wallet like MetaMask needs to do basic things like display your balance, your recent transactions, and your NFTs, as well as more complex things like constructing transactions, interacting with smart contracts, etc. In short, MetaMask needs to interact with the blockchain, but the blockchain has been built such that clients like MetaMask can’t interact with it. So like my dApp, MetaMask accomplishes this by making API calls to three companies that have consolidated in this space.

For instance, MetaMask displays your recent transactions by making an API call to etherscan:

GET https://api.etherscan.io/api?module=account&address=0x0208376c899fdaEbA530570c008C4323803AA9E8&offset=40&order=desc&action=txlist&tag=latest&page=1 HTTP/2.0

…displays your account balance by making an API call to Infura:

POST https://mainnet.infura.io/v3/d039103314584a379e33c21fbe89b6cb HTTP/2.0

{
    "id": 2628746552039525,
    "jsonrpc": "2.0",
    "method": "eth_getBalance",
    "params": [
        "0x0208376c899fdaEbA530570c008C4323803AA9E8",
        "latest"
    ]
}

…displays your NFTs by making an API call to OpenSea:

GET https://api.opensea.io/api/v1/assets?owner=0x0208376c899fdaEbA530570c008C4323803AA9E8&offset=0&limit=50 HTTP/2.0

Again, like with my dApp, these responses are not authenticated in some way. They’re not even signed so that you could later prove they were lying. It reuses the same connections, TLS session tickets, etc for all the accounts in your wallet, so if you’re managing multiple accounts in your wallet to maintain some identity separation, these companies know they’re linked.

MetaMask doesn’t actually do much, it’s just a view onto data provided by these centralized APIs. This isn’t a problem specific to MetaMask – what other option do they have? Rainbow, etc are set up in exactly the same way. (Interestingly, Rainbow has their own data for the social features they’re building into their wallet – social graph, showcases, etc – and have chosen to build all of that on top of Firebase instead of the blockchain.)

All this means that if your NFT is removed from OpenSea, it also disappears from your wallet. It doesn’t functionally matter that my NFT is indelibly on the blockchain somewhere, because the wallet (and increasingly everything else in the ecosystem) is just using the OpenSea API to display NFTs, which began returning 304 No Content for the query of NFTs owned by my address!

Recreating this world

Given the history of why web1 became web2, what seems strange to me about web3 is that technologies like ethereum have been built with many of the same implicit trappings as web1. To make these technologies usable, the space is consolidating around… platforms. Again. People who will run servers for you, and iterate on the new functionality that emerges. Infura, OpenSea, Coinbase, Etherscan.

Likewise, the web3 protocols are slow to evolve. When building First Derivative, it would have been great to price minting derivatives as a percentage of the underlying’s value. That data isn’t on chain, but it’s in an API that OpenSea will give you. People are excited about NFT royalties for the way that can benefit creators, but royalties aren’t specified in ERC-721, and it’s too late to change it, so OpenSea has its own way of configuring royalties that exists in web2 space. Iterating quickly on centralized platforms is already outpacing the distributed protocols and consolidating control into platforms.

Given those dynamics, I don’t think it should be a surprise that we’re already at a place where your crypto wallet’s view of your NFTs is OpenSea’s view of your NFTs. I don’t think we should be surprised that OpenSea isn’t a pure “view” that can be replaced, since it has been busy iterating the platform beyond what is possible strictly with the impossible/difficult to change standards.

I think this is very similar to the situation with email. I can run my own mail server, but it doesn’t functionally matter for privacy, censorship resistance, or control – because GMail is going to be on the other end of every email that I send or receive anyway. Once a distributed ecosystem centralizes around a platform for convenience, it becomes the worst of both worlds: centralized control, but still distributed enough to become mired in time. I can build my own NFT marketplace, but it doesn’t offer any additional control if OpenSea mediates the view of all NFTs in the wallets people use (and every other app in the ecosystem).

This isn’t a complaint about OpenSea or an indictment of what they’ve built. Just the opposite, they’re trying to build something that works. I think we should expect this kind of platform consolidation to happen, and given the inevitability, design systems that give us what we want when that’s how things are organized. My sense and concern, though, is that the web3 community expects some other outcome than what we’re already seeing.

It’s early days

“It’s early days still” is the most common refrain I see from people in the web3 space when discussing matters like these. In some ways, cryptocurrency’s failure to scale beyond relatively nascent engineering is what makes it possible to consider the days “early,” since objectively it has already been a decade or more.

However, even if this is just the beginning (and it very well might be!), I’m not sure we should consider that any consolation. I think the opposite might be true; it seems like we should take notice that from the very beginning, these technologies immediately tended towards centralization through platforms in order for them to be realized, that this has ~zero negatively felt effect on the velocity of the ecosystem, and that most participants don’t even know or care it’s happening. This might suggest that decentralization itself is not actually of immediate practical or pressing importance to the majority of people downstream, that the only amount of decentralization people want is the minimum amount required for something to exist, and that if not very consciously accounted for, these forces will push us further from rather than closer to the ideal outcome as the days become less early.

But you can’t stop a gold rush

When you think about it, OpenSea would actually be much “better” in the immediate sense if all the web3 parts were gone. It would be faster, cheaper for everyone, and easier to use. For example, to accept a bid on my NFT, I would have had to pay over $80-$150+ just in ethereum transaction fees. That puts an artificial floor on all bids, since otherwise you’d lose money by accepting a bid for less than the gas fees. Payment fees by credit card, which typically feel extortionary, look cheap compared to that. OpenSea could even publish a simple transparency log if people wanted a public record of transactions, offers, bids, etc to verify their accounting.

However, if they had built a platform to buy and sell images that wasn’t nominally based on crypto, I don’t think it would have taken off. Not because it isn’t distributed, because as we’ve seen so much of what’s required to make it work is already not distributed. I don’t think it would have taken off because this is a gold rush. People have made money through cryptocurrency speculation, those people are interested in spending that cryptocurrency in ways that support their investment while offering additional returns, and so that defines the setting for the market of transfer of wealth.

The people at the end of the line who are flipping NFTs do not fundamentally care about distributed trust models or payment mechanics, but they care about where the money is. So the money draws people into OpenSea, they improve the experience by building a platform that iterates on the underlying web3 protocols in web2 space, they eventually offer the ability to “mint” NFTs through OpenSea itself instead of through your own smart contract, and eventually this all opens the door for Coinbase to offer access to the validated NFT market with their own platform via your debit card. That opens the door to Coinbase managing the tokens themselves through dark pools that Coinbase holds, which helpfully eliminates the transaction fees and makes it possible to avoid having to interact with smart contracts at all. Eventually, all the web3 parts are gone, and you have a website for buying and selling JPEGS with your debit card. The project can’t start as a web2 platform because of the market dynamics, but the same market dynamics and the fundamental forces of centralization will likely drive it to end up there.

At the end of the stack, NFT artists are excited about this kind of progression because it means more speculation/investment in their art, but it also seems like if the point of web3 is to avoid the trappings of web2, we should be concerned that this is already the natural tendency for these new protocols that are supposed to offer a different future.

I think these market forces will likely continue, and in my mind the question of how long it continues is a question of whether the vast amounts of accumulated cryptocurrency are ultimately inside an engine or a leaky bucket. If the money flowing through NFTs ends up channeled back into crypto space, it could continue to accelerate forever (regardless of whether or not it’s just web2x2). If it churns out, then this will be a blip. Personally, I think enough money has been made at this point that there are enough faucets to keep it going, and this won’t just be a blip. If that’s the case, it seems worth thinking about how to avoid web3 being web2x2 (web2 but with even less privacy) with some urgency.

Creativity might not be enough

I have only dipped my toe in the waters of web3. Looking at it through the lens of these small projects, though, I can easily see why so many people find the web3 ecosystem so neat. I don’t think it’s on a trajectory to deliver us from centralized platforms, I don’t think it will fundamentally change our relationship to technology, and I think the privacy story is already below par for the internet (which is a pretty low bar!), but I also understand why nerds like me are excited to build for it. It is, at the very least, something new on the nerd level – and that creates a space for creativity/exploration that is somewhat reminiscent of early internet days. Ironically, part of that creativity probably springs from the constraints that make web3 so clunky. I’m hopeful that the creativity and exploration we’re seeing will have positive outcomes, but I’m not sure if it’s enough to prevent all the same dynamics of the internet from unfolding again.

If we do want to change our relationship to technology, I think we’d have to do it intentionally. My basic thoughts are roughly:

We should accept the premise that people will not run their own servers by designing systems that can distribute trust without having to distribute infrastructure. This means architecture that anticipates and accepts the inevitable outcome of relatively centralized client/server relationships, but uses cryptography (rather than infrastructure) to distribute trust. One of the surprising things to me about web3, despite being built on “crypto,” is how little cryptography seems to be involved!
We should try to reduce the burden of building software. At this point, software projects require an enormous amount of human effort. Even relatively simple apps require a group of people to sit in front of a computer for eight hours a day, every day, forever. This wasn’t always the case, and there was a time when 50 people working on a software project wasn’t considered a “small team.” As long as software requires such concerted energy and so much highly specialized human focus, I think it will have the tendency to serve the interests of the people sitting in that room every day rather than what we may consider our broader goals. I think changing our relationship to technology will probably require making software easier to create, but in my lifetime I’ve seen the opposite come to pass. Unfortunately, I think distributed systems have a tendency to exacerbate this trend by making things more complicated and more difficult, not less complicated and less difficult.

gm!

Stay in touch,

Moxie Marlinspike >> Blog >> My first impressions of web3

moxie.org | 15 days ago |

230 comments

webview/webview: Tiny cross-platform webview library for C/C++/Golang. Uses WebKit (Gtk/Cocoa) and Edge (Windows)

A tiny cross-platform webview library for C/C++/Golang to build modern cross-platform GUIs. Also, there are Rust bindings, Python bindings, Nim bindings, Haskell, C# bindings and Java bindings available.

The goal of the project is to create a common HTML5 UI abstraction layer for the most widely used platforms.

It supports two-way JavaScript bindings (to call JavaScript from C/C++/Go and to call C/C++/Go from JavaScript).

It uses Cocoa/WebKit on macOS, gtk-webkit2 on Linux and Edge on Windows 10.

Webview for Go developers

If you are interested in writing Webview apps in C/C++, skip to the next section.

Getting started

Install Webview library with go get:

$ go get github.com/webview/webview

Import the package and start using it:

package main

import "github.com/webview/webview"

func main() {
	debug := true
	w := webview.New(debug)
	defer w.Destroy()
	w.SetTitle("Minimal webview example")
	w.SetSize(800, 600, webview.HintNone)
	w.Navigate("https://en.m.wikipedia.org/wiki/Main_Page")
	w.Run()
}

To build the app use the following commands:

# Linux
$ go build -o webview-example && ./webview-example

# MacOS uses app bundles for GUI apps
$ mkdir -p example.app/Contents/MacOS
$ go build -o example.app/Contents/MacOS/example
$ open example.app # Or click on the app in Finder

# Windows requires special linker flags for GUI apps.
# It's also recommended to use TDM-GCC-64 compiler for CGo.
# http://tdm-gcc.tdragon.net/download
$ go build -ldflags="-H windowsgui" -o webview-example.exe

For more details see godoc.

Distributing webview apps

On Linux you get a standalone executable. It will depend on GTK3 and GtkWebkit2, so if you distribute your app in DEB or RPM format include those dependencies. An application icon can be specified by providing a .desktop file.

On MacOS you are likely to ship an app bundle. Make the following directory structure and just zip it:

example.app
└── Contents
    ├── Info.plist
    ├── MacOS
    |   └── example
    └── Resources
        └── example.icns

Here, Info.plist is a property list file and *.icns is a special icon format. You may convert PNG to icns online.

On Windows you probably would like to have a custom icon for your executable. It can be done by providing a resource file, compiling it and linking with it. Typically, windres utility is used to compile resources. Also, on Windows, webview.dll and WebView2Loader.dll must be placed into the same directory with your app executable.

Also, if you want to cross-compile your webview app - use xgo.

Migrating from v0.1.1 to v0.10.0

webview.Open() has been removed. Use other webview APIs to create a window, open a link and run main UI loop.
webview.Debug() and webview.Debugf() have been removed. Use your favorite logging library to debug webview apps.
webview.Settings struct has been removed. Title, URL and size are controlled via other API setters and can be updated at any time, not only when webview is created.
Webview.Loop() has been removed. Use Run() instead.
WebView.Run(), WebView.Terminate(), WebView.SetTitle(), WebView.Dispatch() stayed the same.
WebView.Exit() has been renamed to WebView.Destroy()
WebView.SetColor() and WebView.SetFullScreen() have been removed. Use Window() to get native window handle and probably write some Cgo code to adjust native window to your taste.
webview.Dialog has been removed. But it is likely to be brought back as a standalone module.
WebView.Eval() remained the same.
WebView.InjectCSS() has been removed. Use eval to inject style tag with CSS inside.
WebView.Bind() kept the name, but changed the semantics. Only functions can be bound. Not the structs, like in Lorca.

Webview for C/C++ developers

Download webview.h and include it in your C/C++ code:

C++:

// main.cc
#include "webview.h"
#ifdef WIN32
int WINAPI WinMain(HINSTANCE hInt, HINSTANCE hPrevInst, LPSTR lpCmdLine,
                   int nCmdShow) {
#else
int main() {
#endif
  webview::webview w(true, nullptr);
  w.set_title("Minimal example");
  w.set_size(480, 320, WEBVIEW_HINT_NONE);
  w.navigate("https://en.m.wikipedia.org/wiki/Main_Page");
  w.run();
  return 0;
}

Build it:

# Linux
$ c++ main.cc `pkg-config --cflags --libs gtk+-3.0 webkit2gtk-4.0` -o webview-example
# MacOS
$ c++ main.cc -std=c++11 -framework WebKit -o webview-example
# Windows (x64)
$ c++ main.cc -mwindows -L./dll/x64 -lwebview -lWebView2Loader -o webview-example.exe

C:

// main .c
#include "webview.h"
#ifdef WIN32
int WINAPI WinMain(HINSTANCE hInt, HINSTANCE hPrevInst, LPSTR lpCmdLine,
                   int nCmdShow) {
#else
int main() {
#endif
	webview_t w = webview_create(0, NULL);
	webview_set_title(w, "Webview Example");
	webview_set_size(w, 480, 320, WEBVIEW_HINT_NONE);
	webview_navigate(w, "https://en.m.wikipedia.org/wiki/Main_Page");
	webview_run(w);
	webview_destroy(w);
	return 0;
}

Build it:

# Linux
$ g++ main.c `pkg-config --cflags --libs gtk+-3.0 webkit2gtk-4.0` -o webview-example
# MacOS
$ g++ main.c -std=c++11 -framework WebKit -o webview-example
# Windows (x64)
$ g++ main.c -mwindows -L./dll/x64 -lwebview -lWebView2Loader -o webview-example.exe

On Windows it is possible to use webview library directly when compiling with cl.exe, but WebView2Loader.dll is still required. To use MinGW you may dynamically link prebuilt webview.dll (this approach is used in Cgo bindings).

Full C/C++ API is described at the top of the webview.h file.

Migrating from v0.1.1 to v0.10.0

Use opaque webview_t type instead of struct webview. Size, title and URL are controlled via API setter functions. Invoke callback has been replaced with webview_bind() and webview_return() to make native function bindings inter-operate with JS.
If you have been using simplified webview() API to only open a single URL in a webview window - this function has been removed. You now have to create a new webview instance, configure and run it explicitly.
webview_init() is replaced by webview_create() which creates a new webview instance.
webview_exit() has been replaced with more meaningful webview_destroy().
Main UI loop with webview_loop() inside has been replaced with webview_run() runs infinitely until the webview window is closed.
webview_terminate() remains the same.
webview_dispatch() remains the same.
webview_set_title() remains the same.
webview_set_color() has been removed. Use webview_get_window and native window APIs to control colors, transparency and other native window properties. At some point these APIs might be brought back.
webview_set_fullscreen() has been removed, see above.
webview_dialog() has been removed. But I'd like to see it added back as a separate independent module or library.
webview_eval() remains the same.
webview_inject_css() has been removed. Use webview_eval() to create style tag manually.
webview_debug() has been removed. Use whatever fits best to your programming language and environment to debug your GUI apps.

Notes

Execution on OpenBSD requires wxallowed mount(8) option. For Ubuntu Users run sudo apt install webkit2gtk-4.0(Try with webkit2gtk-4.0-dev if webkit2gtk-4.0 is not found) to install webkit2gtk-4.0 related items. FreeBSD is also supported, to install webkit2 run pkg install webkit2-gtk3.

License

Code is distributed under MIT license, feel free to use it in your proprietary projects as well.

webview/webview: Tiny cross-platform webview library for C/C++/Golang. Uses WebKit (Gtk/Cocoa) and Edge (Windows)

github.com | 17 days ago |

8 comments

Build smaller, faster, and more secure desktop applications with a web frontend | Tauri Studio

Brownfield

compatibility with any front-end framework means you don't have to change your stack

FLOSS

relicensing is possible with Tauri

Bundle

size of a Tauri App can be less than 600KB

Security

is the Tauri-Team's biggest priority and drives our innovation

Patterns

are here to help you choose important features with simple configuration

Cross-platform

compilation allows to bundle binaries for major desktop platforms (mobile & WASM coming soon)

Roadmap

Notice: This roadmap is subject to change.

Released
Ready
To Do
Milestone

CLI
Generate, develop and build Tauri apps from the command line.
Q4 2019
API
Finalize, audit, write documentation and create examples for the smoke-tests.
Q4 2019
Testing & CI
Implement CI with testing and bundle-pipeline validation.
Q4 2019
Desktop Bundler
Bundle for all major desktops from native systems.
Q4 2019
Alpha Release
Technical Release Candidate for desktop, edge cases and bugs acceptable.
Q4 2019
Sidecar
Integrate and instrument other binaries.
Q1 2020
Splashscreen
Use a splashscreen while the main content is loading.
Q1 2020
App Storage
Use a canonical location to store userdata.
Q2 2020
Native Notifications
Cross-platform notifications using polyfilled WEB API.
Q2 2020
GH Action for Building Apps
Build your Web application as a Tauri binary for MacOS, Linux and Windows
Q3 2020
VS Code Extension
Commands and validate tauri.conf.json
Q3 2020
Core Plugin System
Build reusable plugins to extend Tauri core.
Q3 2020
CLI Updater
Update core dependencies automatically from the CLI.
Q3 2020
Webview Bindings
New pure Rust Webview bindings.
Q1 2021
Keyboard Shortcuts
Hook and react to keypresses.
Q1 2021
Multi Window
Run multiple window instances in Tauri.
Q1 2021
Rust-based CLI
Create Rust CLI.
Q1 2021
Transparent Window
Have transparent windows.
Q1 2021
Secure Context for Web APIs
Setup secure context to enable access to restricted APIs.
Q1 2021
Self Updater
Update Tauri Apps from within Tauri.
Q1 2021
Window Menus
Desktop Cross-platform Window Menus.
Q2 2021
App Tray
Desktop Cross-platform Icon Tray.
Q2 2021
Beta Release
Generally stable on Desktop, API locked down.
Q2 2021
Channel API
Send messages through a channel.
2021
Security Audit
3rd party security audit of core libraries.
Q2 2021
2021
Iframe with native API
Opt-in feature enabling for iframe dialog with Tauri and the native API.
2021
Scoped FileSystem
For improved security of files interaction.
2021
Clipboard
Enable programmatic and keyboard access to clipboard.
Planned
One-Time Commands
Run a command that is no longer available after first run.
Planned
DENO Bindings
Use Deno to build your App's backend.
Planned
Mobile Bundler
Bundle to all major mobile device operating systems.
Planned
Cross Compiler
Generate bundled binaries from select operating system environments.
Planned
PureOS App Store
Verified builds for PureOS.
Planned
Stable Release
Stable on On all Platforms.
Planned
Other Bindings
Go, Nim, Python, C++ and other bindings are possible with the stable API.
Planned
Alternative Renderer
Candidate presentation for Webview Alternatives, including GL windowing.
Planned
Tauri-Frida
A decompiler and threat analyzer for Tauri Apps, using Frida.
Planned
The Future
Something missing? Got a great idea? We want you to help us make it happen.
& BEYOND

Build smaller, faster, and more secure desktop applications with a web frontend

tauri.studio | 17 days ago |

340 comments

About Me | DC Rainmaker

Hello!

Thanks for stopping by (or re-stopping by)! Here’s some background on who I am, and what this blog’s all about. I’m just a triathlete and runner like you: I run, I bike, and I do something resembling swimming. And at the end of the day…I blog about it. But other than that last detail – I’m just like you (unless you blog too, in which case we’re the same).

I grew up in Seattle, Washington and ski raced pretty competitively as a kid. Training for that somehow led to running cross-country my freshman year of high school. After my freshman year I decided that running was a miserable way to spend my time, and proceeded to not run for another 10 years or so (really, no running!).

During those ten years I used to my spare time to get pretty into technology and travel. I’d eventually end up graduating high school straight into the world of technology consulting for large organizations (and travelling up to 52 weeks a year). And until recently, that was what I did full time – design computer networks and systems…and still travel a lot. A real lot.

My Return to Running!

Before we get to today – let’s jump back a bit athletically, to the spring of 2006, when I started running again. By ‘started running’, I mean I ran one mile and nearly collapsed. Ok…so I may not have made it that first mile…but we won’t get too caught up on details.

About 4-5 months later I went onto finish my first marathon – the Marine Corps Marathon in Washington DC in 4 hours and 23 minutes. Oh…and along the way in that effort I lost about 60 pounds…

The Start of Triathlon…

I ran through the winter following MCM and when spring sprung I got the bright idea to try and finish a triathlon. Not thinking it through entirely (who really does anyway?), I simply signed up for a half-Ironman triathlon later that summer.

The fact that I hadn’t actually swam a single lap in…ever…should probably be overlooked. Poster-child for how to properly train and sign-up for a triathlon I am not.

So…into the pool I went. I then dusted off my old mountain bike and got to work (the one I used as a kid delivering newspapers). A few weeks later I realized that perhaps my mountain bike wasn’t ideal for a half-ironman – and upgraded to a basic Fuji Road bike. And about two months following that I finished my first triathlon – a sprint, before going on to finish my second triathlon (an Olympic distance event), and finally a few weeks later – The Big Kahuna – my first Half-Ironman (all without aerobars I’d point out!).

Oh, and around that time I started blogging. After that I was hooked (on the triathlon, and apparently, blogging too)!

Getting faster and going longer!

Even prior to completing my first half-iron triathlon, I had already kinda-sorta signed up for my first Ironman. Thankfully, the half-iron worked out.

Thus, the following year with the help of a local coach – Coach Alan Melvin – I went onto complete my first Ironman Race – Ironman Canada. In addition, I also raced a number of other half-iron and shorter distance races.

When I got to the end of the year, we refined my schedule and I ran the Philly Marathon in a time of 2:54 – slicing off 90+ minutes from my marathon time just two years prior. Apparently, training right can get you faster.

Since then, I’ve been sprinkling in Ironman races (Canada and Florida), marathons (like Boston!) as well as a ton of shorter and far less painful triathlon and running races. I enjoy them all. Well, actually, I really enjoy the sprint triathlons…it’s sorta like playground crack.

Look ma, I blog!

Naturally, as I started blogging the same day I signed up for Ironman Canada (the first time), my little place here on the internet was mostly my personal training log. Along the way I also wrote about random things like cooking and the sort. Basically, whatever floated my boat.

At some point I ended up posting to the blog an older e-mail I had written for some co-workers giving my impressions of a running watch (the trusty Garmin FR305 – little red!). That in turn started my foray into product reviews. Which, is likely how you ended up here today.

Going all in!

In October 2015 I decided it was time for a change. So after 15 years in the IT world, and millions of miles flying for work – I returned my work laptop and left my day-job. I suppose that makes me a professional blogger, or perhaps professional writer, maybe even a professional athlete (after all, I have to swim/bike/run to make this work!). But, hopefully, it doesn’t make it any different than before. I’m just going to keep on doing what everyone has hopefully enjoyed till that point I’m going to keep on, keepin’ on.

But, it’s not just me.

In addition to the blog and training, I also have a beautiful wife and family. My wife, “The Girl” and I met at the Pacers running club just outside of Washington DC, and have been training and racing together since. In fact, readers of the blog got a bit of a front row seat through all of that.

After we got married, we decided to spread our wings and enjoy some big world adventures. We packed up our DC home, and crossed the Atlantic to reside in Paris for some five and a half years. In our time there The Girl opened her first Cupcakery and then a second Cake Studio. You can read all about our Parisian adventures in the Paris blog.

Certainly not to be forgotten, we accumulated more family members as the years continued into our Expat life. Please understand, when I say “family members”, it’s simply due to a growing need for more product testers 🙂

After the birth of our second daughter, we decided it was time to bid adieu to the city where we had made such amazing memories and life long friends. Life is short, and there was another city tugging at us.

We moved to Amsterdam, and for the moment we have settled in here. We built out the new DCR Cave, and I officially have The Girl on my team to keep all the behind the scenes organized- so I can focus on getting all of you readers the best and up to date info.

Welcome to the blog – and thanks for reading!
-Ray

About Me

dcrainmaker.com | 17 days ago

Unscatter, What Is It? - The Unscatter.com Blog

Welcome to Unscatter.com, what is it?

I think perhaps the best way to put it is, Unscatter.com is a news discovery and search engine that cares about your privacy. You can check it out if you have not already - Unscatter.com

I think the other thing to point out before I get any deeper is, Unscatter is a personal project being developed by one guy in his spare time. Currently it has no revenue generation, however I would love for it to one day at least pay for itself. Current budget for hosting and Cloudflare is just under $50 a month.

Ok, but what is it?

Unscatter.com is a searchable index of the stories people have been sharing with each other for the past 30 days. By default search results will only include stories shared in the past day. There is an option to expand your search to last week or the last 30 days.

Stories are currently sourced from Reddit.

What about that “cares about your privacy” statement?

Unscatter doesn’t track you. I don’t use cookies and I don’t even log the ip of visitors. It’s dropped from the access logs.

Unscatter does use CloudFlare for CDN purposes, so ideally we don’t even see most requests. CloudFlare was chosen mostly for cost, Unscatter is completely self-funded right now. That means if traffic grows, most requests won’t even make it to the site.

I depend on Cloudflare for analytics. The only self-hosted analytics I do is capture the term searched for every search. It’s a 2 field table, term and timestamp. This actually was added 5/1/2021 and I have not built anything to use that information yet. I do plan to start dropping everything older than a month there as well.

The full privacy policy is posted here https://www.unscatter.com/privacypolicy.

Why only Reddit?

Truth is, I would love to get more sources. I think only pulling data from reddit means I am pulling from the reddit communities echo chamber, especially because I’m letting reddit tell me what the top 100 stories are.

My goal is to eventually try to find more sources. I started with reddit because their API and PRAW SDK made it extremely easy to get a data set I could work with.

I am currently working on integration with Twitter to add it as an additional source for content. I’m open to suggestions for more sites to integrate with, feel free to hit me up on Twitter @joerussbowman or @unscatter.

I think the one important distinction to make is the focus of Unscatter is I’m trying to find and rank stories people are sharing. I’m not interested in finding and crawling all the news sites on the Internet to end up with a database of stories people may not be interested in. Sourcing from link sharing sites and social media is the goal.

What technology is Unscatter using?

I’ll do more articles later on the technology I’m using for Unscatter. I’d say one of the primary purposes of the site has been to create opportunity for me to learn new things and practice existing skills.

The database for Unscatter which also does the full text search is PostgreSQL. I recently discovered CouchBase, I may try it out to see how well I like it compared to Postgres, but I haven’t spent enough time researching it yet.

The Unscatter agent is written in python. This was a no-brainer decision, it gives me immediate access to PRAW for working with reddit and also spacy and newspaper for text analysis.

Unscatter.com, the website is apache acting as proxy for a custom application written in Go.

And I’d like to point out Let’s Encrypt was the final push to getting me started building the site. When I had been dreaming about building the site in the past paying for SSL certs was always one barrier to getting started.

Wrapping up

I think this is good for a first post. I’m open to questions and comments. Would love to hear any suggestions and hey, if you find this site at all useful I’d really love to hear that too. I use it every day, I post stories I find on it regularly to Facebook, Twitter and Instagram.

Thanks for taking the time to get all the way to the bottom of this blog post.

Unscatter, What Is It? - The Unscatter.com Blog

blog.unscatter.com | 17 days ago

Ask HN: Let's build an HN uBlacklist to improve our Google search results? | Hacker News

		Ask HN: Let's build an HN uBlacklist to improve our Google search results?
		162 points by sanketpatrikar 2 hours ago \| hide \| past \| favorite \| 94 comments
		For the unaware, uBlacklist [0] is a browser extension that lets you blacklist sites from the google search results page. It lets you blacklist sites right from the results page, by regex, or by linking lists hosted somewhere. The low quality of results has been a problem from a while now and has become worse lately thanks to all those StackOverflow and Github clones. So I was wondering if we could come together and contribute to a single blacklist hosted somewhere and then import it into each of our browsers. Who knows? We might end up improving the quality of the results we all get. Lists to get rid of the StackOverflow and Github clones already exist. [1] I would love to contribute to a project like this, but won't be able to be a maintainer due to time constraints. Would greatly appreciate it if someone could host this. A simple txt file on github would do. What do you say, HN? [0]: https://github.com/iorate/ublacklist [1]: https://github.com/rjaus/awesome-ublacklist

Use a search engine that doesn't fuck up your results instead of trying to unfuck the results it gives you. Why go through this much effort to still give your money to google?

My experience with ddg has been very positive over the years. Although I think my eyes have gotten good at skipping over the noise in the search results. It might be nice to offload that effort to software.

Just tried your search in both Google and Duck Duck Go. On Google first page spam copies are ~80% of the links, on DDG maybe 40%. Not good, but much better than Google.

That's an ambitious goal, I'm not sure to see how that would be maintainable on the long run.

On a much smaller scale, if anyone is interested, I maintain a black list focused on those code snippet content farms that gets in the way when you're searching for some error message or particular function here https://github.com/jhchabran/code-search-blacklist.

Isn't this Google's job? Are developers a small but lucrative target and so the suits at Google don't see the benefit of improving that experience by cleaning up the spam?

Can we just nudge them to do so under the threat of an influential minority leaving due to their use case being affected?

Is there a word for this tendency to say, "it's someone else's job" as a justification for doing nothing at all to help or improve one's own circumstances? I see more and more of it in the public discourse over the last years and it kind of bothers me. I see it a lot in conversations related to poverty or climate change, but it is as we see here by no means exclusive to those topics.

To the original replyer: you could wait for Google to do something, but if they were going to fix the listicle issue, and it were fixable on their end, they'd probably have done it by now. I'm disappointed in the situation too but if there is a workable solution on our end it would be silly to ignore it because fixing the problem is someone else's job.

To the OP: I worry that the number of domains pumping out crap might be far greater than we know, and that might hamper the effectiveness of this. If the collaborative block list ever got big enough you might also have to deal with spam. But I think it would be a great thing to try. This is one of those issues that annoys me, but it's just below my action potential threshold. My biggest objection right now is the spammy recipe websites.

> Is there a word for this tendency to say, "it's someone else's job"

At the risk of sounding pretentious, I call it "socialist", since they spend their lives telling others what to do or what is good or not for the rest of us but they rarely do anything about it. Surprisingly, this is the group that is really worried about poverty and climate change and do as much as I do for it, with the difference that I do it by myself, the few times I do it, not requiring the rest to do it.

It is always someone else who will do it. Though the other day I had a conversation with a non-socialist person that had that same attitude ("other should do it") towards what OTHERS should do. I really dislike that attitude, no matter where it comes from.

Point at hand: when I want or promote something, I am the first one to do it no matter others do it or not. The rest, no matter the ideology, all b*llshit.

As imperfect as I am, I try to do what I think is good (and sometimes my imperfection prevents me from doing it) but I do not spend my life telling other people why they are worse than me and telling them what they should do or not. The most I have for someone is good suggestions, never requirements.

This has nothing to do with socialism. Since it is a problem under capitalism why don't we call it "capitalist". Or maybe we could just say lazy.

I think the question was specifically about declining to improve one's own circumstances because someone else could, not declining to help someone else. That's different from the bystander effect as usually conceived. The theoretical bystander effect is something like "Someone is being attacked, I could call 911 but someone else will;" this is perhaps something like "I am being attacked, I could scream for help but other people will notice anyway" - but really more like "I can't cross the street in front of my house because the storm drains by the crosswalk are clogged, I could rake them but the city is supposed to do that."

I suggest this because there can only be so many websites that use SEO to game their way to the top and bury the good results beneath them.

If we manage to block them, we might be able to get a results page with good sites upfront and the other meaningless content below it. I assume Google will also surface good content along with the bad, so our blacklist might enable the good stuff to reach the top.

The spam problem, I'm sure of yet, but we might either be able to block enough of it to be satisfied or it won't pose a problem for most searches that are currently giving bad results.

Thanks, that's a good thing to remember. The other concern is installing ublocklist -- I'm scared to give more extensions access to google.com. I wonder if I could fork it and restrict its permissions to the SERP.

>” Can we just nudge them to do so under the threat of an influential minority leaving due to their use case being affected?”

I sense Google is too big to cater to us like this. Despite a steady decline in quality, Google is still the dominant search engine and the competition isn’t even close to its market share. Not only would they not notice many of “us” leaving, the amount of change they would have to implement in order to satisfy our desires would end up changing the product for the rest of the market. On some level, the product managers must be satisfied with the metrics as they stand since Google is continuing with their current course.

> Isn't this Google's job?

Have you searched anything on Google lately? The answer is "no". Their new job seems to be to stuff your results with anything even remotely related (and sometimes related in a way that only machine learning can see) so you have things to click on.

Edit: with the lone exception of "find me this bussiness nearby".

If I search for a business name its normally not the first result any more. Usually an advert for another company in the same space I dont want to use (basically an offensive / scamming result from user perspective) and then also the standard "buy search term on amazon".

I think you're wildly overestimating the influence of the minority within HN (or other similar communities) that actually care enough to switch to another search engine.

This reminds me of the Linux gamers who claim that they can influence game development companies by purchasing games with Linux ports, but wind up being less than 0.5% of sales of most games with Linux ports, which leads manufacturers to ignore that customer base almost completely.

Not disagreeing with you.. big companies truly mostly ignore Linux but there are more than a few indie devs who support Linux as a platform. And I tend to play only indie games these days anyways because all the big commercial games have been reduced to some kind of click-and-succeed or free-to-play-and-milk-some-whales crap.

I personally am kinda happy where Linux gaming has come to be. Sure it could always be better but I remember times where there were only like 3 games for Linux and you had to compile them yourself..

game companies might have ignored us, but in the end it created a space for valve and codeweavers to fill.

It is Google's job, but they either aren't doing it or are failing at it. We could do something about it at least until a better alternative or a solution appears.

> Can we just nudge them to do so under the threat of an influential minority leaving due to their use case being affected?

Many influential people have tried and nothing seems to have transpired from it.

Google.com is the most popular website. I don't think the leaving of any minority group we manage to create would even matter to Google, let alone force them to fix the issue. Not that I discourage using alternatives.

> cleaning up the spam

I'll be happy to be proven wrong, but I think Google is now fully in the 'optimize for engagement' camp. If that's what they're doing, it's by definition not spam (from their point of view) if people are clicking on it more than the non-spam results.

Again, only my guess as to what's going on. I don't see another good explanation for them only serving cloned Stackoverflow and top X lists for basically everything now.

From a user point of view, a search engine’s job is to link you away from the search engine, so how does a search engine measure engagement? Is it time on page or maybe number of searches performed? When you don’t have viable competitors both of those are improved by worse search results. Even number of ads clicked would be improved with worse search results because ads don’t have as much competition for your attention when there are no relevant results on the page.

> From a user point of view, a search engine’s job is to link you away from the search engine

That hasn't been true for a long while.

Users are there for an answer to their query, not to be directed anywhere. Frequently this comes in the form of search vertical response that doesn't lead the user anywhere.

From Google's point of view, you want the user where you can show them ads, full stop. Google does not exist to provide you with answers to your queries, it exists to make money for shareholders.

> When you don’t have viable competitors both of those are improved by worse search results.

I don't think "worse search results" is the way to think of it. From G's point of view they're better because they make them more money.

>Isn't this Google's job?

Or more fundamentally perhaps this is just the system working as Google intended?

I think the disconnect comes from people expecting perfect search results as curated by humans, whereas Google necessarily must optimize for automated results. Automated results will never be perfect.

Instead of spending energy to change Google, why not just leave them for good?

Start with changing default search engine to DuckDuckGo or something else, install uBlockOrigin and Privacy Badger to disable tracking, and gradualy reduce using every Google or application, starting with Chrome.

Be the change you want to see.

I relate to this opinion. There are two reasons why my suggestion might still be useful:

1. DuckDuckGo too is affected by these SEO-gaming sites, so maintaining a blacklist will help us make that experience better too.

2. There are times when only Google can find us what we're looking for, so this will prove useful when we go back to it.

I'm sure they have great books on stackoverflow answers, reddit reviews of products and opening times of local stores.

Google has a fiduciary responsibility to shareholders, which is so much work as it is! Why are you trying to ask them to do more?

I am keenly interested in this idea.

I sense that in the near future the paradigm of search engines will go from the current “index everything and become a universal answer engine” to “index a small subset of the Internet and become an answer engine honed towards a specific topic/domain”.

Thus could be really helpful for us at Mojeek, and generally helpful for all search engines we think, done as a list rather than as an extension.

This looks like a big, time-consuming project that would rely on a private Google API that can change any time. I think it's not worth to invest your effort into that. I wish more people would help to improve FLOSS, peer-to-peer search engine YaCy instead, https://yacy.net.

This improves other search engines as well, not just the Google universe. I'm sure even an opensource, peer-to-peer search engine will have similar issues of content farm content and gamed pages if it becomes large enough to compare with search engines like DuckDuckGo.

On the other hand it is absolutely ridiculous to conflate the difficulty of occasionally adding a domain to a local filter, and assisting to build a random unproven search engine. People volunteer their development effort for projects they personally find interesting or challenging. If you want more developers advocate for the project don't try to scold people for wanting to spend a small amount of their time refining a solution that works for them.

Why not use another search engine, such as Kagi, which has built-in support for this? At least for the programming niche, Kagi has worked really great for me for a month now.

This is just my own personal preference, but I manage my own list of what is blocked or allowed on my systems. I would be concerned that a group contributed list for this category of blocking could quickly devolve into a group-think censorship dominated by whomever is the most devoted to blocking and extending echo bubbles to peoples browsers.

Seeing "how other people configure their tools" can be interesting. I love seeing how people configure their .bashrc with custom commands.

I don't think I'd want to download a list of the most blocked sites and plug it into one of my tools though, for some of the reasons you mentioned.

Sounds like we'd just need an HN uBlacklist subscription. Sourcing and validating submissions to the blacklist is the problematic bit. Perhaps use HN as an OAuth provider (not currently an available feature), use rules based on account age and karma for allowing or scoring submissions, voting system like HN ... sounds like something that might actually do better hosted on HN.

My obligatory 2cents that we better stop calling it "blacklist" due to obvious reasons.

Proposed alternatives for blacklist/whitelist include:

denylist/allowlist blocklist/passlist

It's usually better to support efforts that have a greater chance of succeeding, than supporting the ideal solution that is unlikely to happen.

...assuming "support" is a substantive action and thus a finite resource, and not just an upvote on a social media page.

Huzzah, that extension supports other browser engines as well. It's not nearly as atrocious an issue on DuckDuckGo but there are still some of those re-post heavy sites that aggressively get through, as well as some low quality content farms. It's nice to have a tool available to do local/personal fine grained refinement.

Google's existing blacklist is half the reason (for me) that the results quality has declined so much. So this is not a great solution.

Google has precious little incentive to filter out ad-ridden spam content, given that their entire business model is the very ads these sites are plastered in.

In short, if you find what you are looking for, they get few ad impressions and make less money. Meanwhile, if you have to click through half a dozen spam results first, there are many ad impressions, and they make more money.

Are you sure it hasn't been (from their point of view)? I've got that old familiar feeling that there has been a change internally about what now constitutes a "quality" search result. Google has been moving toward optimizing for engagement over what we normally thing of as relevance for a while.

I imagine that Google's incentive is to first prioritize results that have the most profitable Google Ads on their page, and then for quality of result, not the other way around.

Until that incentive structure changes, Google will not be interested in solving this.

Google is a lost cause by now. The only solution is a competitor. DDG is just as bad since they source results from google so that is not the solution.

DDG don't source any results from Google. They rely on Bing and increasingly their own bot.

> and increasingly their own bot

Except they so obviously don't. They lie through their teeth and their good at marketing, that's it. They've zero technical ability.

A reply to those in this thread saying that Google should/will take care of this:

Imagine you had a position in a huge market that was as close to unassailable as there has ever been. Imagine also that you have a controlling position over the mechanisms that allow people to participate in that market.

Now try to make a case against optimizing for squeezing every last cent out at the cost of the user experience.

In 10 years we will regard Google the way we regard cable companies today. Maybe even worse since we need to be able to search for answers more than we ever needed cable TV.

I'd love to contribute. If my small contributions collectively have the potential to save many hours of others' time, it may end up the most impactful thing I do this year.

Wouldn’t it be easier to simply invert the problem and come up with a whitelist instead? When searching for technical info there are only a handful of sites I use, namely stackoverflow and wikipedia.

I just use duckduckgo with fallback bangs when the results aren't satisfactory to me, usually falling back to Bing.

Google doesn't deserve my time or my eyes.

TL;DR: I strongly suspect that relatively small, personally-curated lists will be much more appropriate and highly effective. These might be augmented with specific classifications, but probably not on a widespread basis.

Though the proposed solution borrows heavily from concepts long used in email and Usenet spam, there are a few critical distinction in SEO SERP[1] spam which both make a widely-crowdsourced listing less applicable and less necessary.

In the case of email, your inbox is an unlimited resource to the spammers --- there's effectively no limit to how much spam they can throw at it. As there are also an effectively limitless set of source addresses (by either domain name or IPv6 addresses), and because email/Usenet spam is itself a quantity/numbers game with rapidly shifting origins, collectively-source and curated blocklists have value.[2]

A SERP is itself a finite resource --- the default is to display 10 results, and not making it into the top ten provides little reward. Moreover, high ranking search takes some effort and time to achieve, it's not like in email where a new server can spin up and immediately start deluging targets.

My experience with annoyances matching this sort (stream-based social media is one example) is that blocking a relatively small number of high-profile annoyances hugely improves signal/noise ratios. And I think that will be the case with SERPs as well. There are a half-dozen or so sites which tend to dominate results in most cases, and those can be individually blocklisted (if the capability exists). If more appear, they can similarly be removed.

The other factor is that quite a few sites which some people find exceedingly annoying and spammish, others find appealing. Coming to agreement on what to block, and classifications of such domains / sites, is likely to be difficult and/or contentious. There may be exceptions in specific instances (hence: specific classifications of unwanted results), but less so in the general case.

I might be wrong. The case of DNS adbloc, with PiHole as the classic example, shows that very large lists can be compiled and used. My own Web adblock / malware block configurations have typically had from ~10k to ~100k of thousands of entries. That said, the really heavy lifting is typically done by a much smaller fraction of the total. Power laws and Zipf functions work in your advantage here.

________________________________

Notes:

1. Search engine results page, that is, what you see in response to a query.

2. Even in the case of email spam, the principle value is largely from curated lists, usually by experts, e.g., Spamhaus.

In that case, we'd still need a repo bringing together all these individual lists. I couldn't finding anything like this.

I suggested a single list to prevent repetition and to limit the imports one needs to make to one.

Im afraid this is potentially dangerously political.

Are you only going to filter obvious spam and sites that republish other’s content, or are you going to block sites that are “harmful” or disseminate “disinformation”?

Who will get to decide which media bubble I’m in?

This is a browser extension that you have to explicitly install and configure. You get to decide which media bubble you are in (setting aside the fact that google may or may not curate their search results.)

It can be made configurable - an option for github clones, another for SO clones, etc..., the folks who want to filter "harmful" content will add their category

Who sets the default 90% of people won’t touch?

Someone, give me the late 90’s cesspool of un-curated internet back! Everything is so… trite now.

Nah. Blocking isn’t the answer. What we need is a better search index.

I find it difficult to believe that relatively beginner NLP projects get posted here all the time, yet no one has adapted that stuff to create a new search index.

Personally I don’t know enough to really do this well, but I can tell just blocking sites from Google’s results isn’t the way.

With the consequence that the default branch now differs between locally and remotely initiated repositories

For old versions of git, yes.

However, recent versions come with "main" as the default.

And since 2.28, released more than a year ago, we have the option to change the default:

    git config --global init.defaultBranch main

Whitelisting and greylisting is still fine. The current movement (primarily in the USA, but not only) to replace the word black where the meaning is associated with somewhat negative is quite superficial and happy with simple swaps. And this is quite good, because if we went deeper, we would have many problems related to the fact that many positive values are associated with light and many negative with darkness. So people would either have to rewrite the fundamentals of our culture (and maybe even biology), or to accept the fact that these two - the pigments in our skin, which is neither white nor black btw, and the culturally-influenced meanings of various colors - belong to two different semantic planes.

Ask HN: Let's build an HN uBlacklist to improve our Google search results?

news.ycombinator.com | 18 days ago

Show HN: No Trash Search | Hacker News

		Show HN: No Trash Search (notrashsearch.github.io)
		125 points by rickdeveloper 1 day ago \| hide \| past \| favorite \| 86 comments

I built this website a couple of months ago because I was annoyed by how hard it was to find useful things on Google. As "Google no longer producing high quality search results in significant categories" [0] is currently #1 on the front page I figured I'd share this project again. I hope it's useful to some people.

'No Trash Search' is very focussed on STEM and not "for daily use". It's surprisingly good when you're looking for certain kinds of information. Under the hood it's little more than a programmable search engine [1] with a whitelist of ~120 sites.

[0] https://news.ycombinator.com/item?id=29772136

[1] http://programmablesearchengine.google.com

> Under the hood it's little more than a programmable search engine [1] with a whitelist of ~120 sites

So back to what web search was in the 1990s, roughly: an index from a curated selection of sites.

While I can understand the appeal, restricting your search engine to only ~120 websites out of hundreds of millions (?) is basically giving up on the Web.

(BTW, any good search engines these days that aren't indirectly using Google or Bing ?)

> restricting your search engine to only ~120 websites out of hundreds of millions (?) is basically giving up on the Web.

Sure - the web is now a cesspool optimized for advertising and attention. The traditional search engines made a lot more sense at the dawn of the internet when it was more about discovery. Now, for the most part, it's closer to an information retrieval tool, where a finite list of established sites have the bulk of what one is looking for. It only makes sense to have a tool that lets one navigate the established, legit internet, and not have to deal with all the crap.

That doesn't mean there is no use case for google as it is, but some more focused competition is a no brainer.

There's http://yandex.com . It's great if you want to search controversial subject matter and controversial results that Google wouldn't give you. The reverse image search is also amazing.

The reverse image search in particular is very, very good.

Far better than Bing or Google. It's not obvious why theirs is so terrible, unless that product is not a moneymaker for them, in which case it explains everything.

Most. Yandex is great, especially for programming searches. It generally ranks GitHub, Stack Overflow and other content-heavy sites highly. Google has been taken over by weird clones of GitHub and SO lately, Yandex has no such trash.

It completely boggles my mind that the useless GitHub and SO clones rank first page on Google. Do engineers at Google not use their own product?

Regarding stackoverflow there is a fair chance they can congratulate themselves:

If I am right they played stupid games and won stupid prizes. More specifically they have allowed rampant deletionism for years so while I am fairly certain the questions and answers originated on Stack Overflow it wouldn't surprise me if a good number of of those aren't visible on Stack Overflow anymore which would explain why they rank higher in Google.

Done right this would actually be a service.

Sadly some of them seems to mix together various questions and answers in the same page to generate text matches for unusual queries.

Frankly, no. It's kind of a running joke that you can't Google any of your problems at Google because everything is internal.

Don't have time to mess with it right now, but does it normally return about half results in Russian or is that something my phone/browser is doing?

> Google has been taken over by weird clones of GitHub and SO lately

Do you have an example search leading to a GitHub clone?

French is my mother tongue, but I've quickly learned during my studies that using English keywords in my STEM-related searches would simply lead me to better (and more abundant) results.

A few weeks/months ago however, while I was trying to solve an issue whith a colleague who would search using french keywords, I noticed that some websites featured on the first page of the Google results were off.

In short, they were machine-translated versions of Stack Overflow threads. And they would appear in most of the searches using french keywords.

Those websites also appeared rarely in my searches while I was using English keywords, but most of the time I never bothered opening them. But now I notice them every time.

Some examples: When searching for "wget set http proxy" on Google, the fourth result leads me to qastack.fr, and the ninth to it-swarm-fr.com, both are websites featuring scrapped and machine-translated threads from Stack Overflow.

When searching deliberately in french for "Eclipse CDT stdout ne s'affiche pas" ("Eclipse CDT stdout not displayed [in console]"), the first result leads me to askcodez.com and the fourth one to qastack.fr (askodez is the same as the other two).

I have never stumbled upon Github clones, yet, however.

I read that you can achieve similar results with uBlock Origin, but I indeed ended up using uBlacklist as you can block websites on the go and because it supports multiple search engines. It is working perfectly, one more reason to use FF I guess.

yep it's always a bunch of movie subscription sites instead of the torrent. it's almost like Google's search engine is predominantly focused on collecting advertising dollars...?

"(BTW, any good search engines these days that aren't indirectly using Google or Bing ?)"

The code for Gigablast is open-source, including the crawler.

I could be wrong but I do not think search.marginalia.eu nor wiby.me use Google or Bing.

The comment about "hundreds of millions" is interesting. Assume hypothetically a search engline claimed to be searching millions of sites for a given query but in truth it was actually only searching 120 sites that it had determined answered this query (i.e., was the most popular answer source) for the majority of users. How would a user verify the search engine's claim about searching millions of sites was true. What if the search engine only allowed the user to retrieve a maxmimum of about 230 results, not matter how many sites it claimed to search.

"How would a user verify the search engine's claim about searching millions of sites was true."

Search for things specifically on those pages, by very specific phrases and such.

Of course you have to find them yourself first for that verification.

I can say having set up some very teeny tiny websites here and there that the googlebot is hooked up to a lot of stuff. I'm not even sure how it found a couple of them as quickly as it did. Things like "if someone adds an RSS feed to Feed.ly" seem to do the trick. None of them were sites trying to "hide" or anything and I expected them to be found eventually, but they got found much faster than I expected. Or maybe they just scan new domain registrations, though it seemed to me it wasn't that that triggered it.

Imagine searching for something that is quite common that will produce a large number of results but the user can only retrieve, say, 230 results total. How does the user verify that all of the "millions of sites" that contain results were actually searched when the user submitted her query.

A search engine can tell users some large number of sites were searched at the time of the user's query and some large number of results exist, but what if it does not allow the user to actually view all the results.

To put it another way, the question is not what Google has discovered about the www,^1 but what Google is willing to let the user search and retrieve. If retrieving the 963rd result for a common string is not allowed, then it is impossible for the user to verify that the site containing that result was searched when the user submitted her query. Even if the search produced a 963rd result, what difference does it make if the user cannot retrieve it. What is the point of the search engine locating the 963rd result if it never has to show this result to the user querying a common string.

1. What Google has discovered about the www^2 and what Google users are able to discover about the www through Google may be two different things.^3 Google has its own interests to pursue in the name of online advertising and these may conflict with users' interests. "Censorship" is one concept that often draws negative connotations but there are many more subtle forms of filtering and manipulation that are possible here, including unintentional ones.

2. The most important focus would be what is "popular".

3. Some users might care less about what is "popular". Such users would, by and large, be less interesting to an advertising company. Individual interests might become subverted in favour of "popular" interests, to the extent they conflict. An advertising company (that runs a search engine) will favour the larger audience.

All the search engines have trash. I retrieve results from a variety of search engines and mix them into a simplified SERP with zero cruft that can be read very quickly. Some call searching multiple search engines "meta-search". The main differences with mine is 1. it is all done client side (there is no remote "meta-search" engine) and 2. searches can be "continued" where they left off at any time. This allows one to avoid rate limits. There are always trash results, every search engine has them in their SERPs, but I find that the more results and the more varied the results the better the chance of finding useful, non-trash ones. Gigablast allows returning at least 100 results at a time. Few search engines allow 100 results at a time that anymore. Google still allows it but will not allow a user to retrieve more than 200-something results total.

Yeah I do my own crawling, and offer results from around 200k sites (although it's indexed 700k domains, most of which are crap).

I need to buy a 3 hole punch, and when searching for reviews yesterday I had the same problem of lots of hits with affiliate links and low quality sites.

I searched for “3 hole punch review” [1] here, and the results have zero relevancy.

First one is a Chinese cell phone company, second a Wikipedia page for an episode of the office, third a thesaurus page with synonyms for ‘colorful’ and fourth a link to the Wikipedia page of Yellow Submarine.

I can’t even imagine how you get there from “3 hole punch review”

[1] https://notrashsearch.github.io/?q=3+hole+punch+review

Now your comment above is the first search result for "3 hole punch" on NTS!

Nice SEO campaign ;)

If only I could get NTS to whitelist my domain name (myfirstnamelastname dot com), the Big-G has hated it seemingly since even before I acquired it > ten years ago, even though it's ad-free and totally benign. Good thing I mostly just host go pkgs with it and use it for my email.

p.s. OP this is amazing! Would love an article explaining any backstory and details on how you made this (or setup / configured it).

> If only I could get NTS to whitelist my domain name

There is a form [0] on the about page that allows people to suggest websites to add :)

> p.s. OP this is amazing! Would love an article explaining any backstory and details on how you made this (or setup / configured it).

Thanks! I think this is gonna be disappointing from an engineering perspective, and certainly not article worthy :) As further explained in my other comments, the website is basically a wrapper around google programmable search [1] where I whitelisted a set of sites I found useful personally, plus some suggestions from other users. It's really easy to set up.

As to why, I will quote some other comments of mine:

"I built this website a couple of months ago because I was annoyed by how hard it was to find useful things on Google."

"to find things more easily while programming or studying (I study biology, cs and ai; and philosophy in my free time, so expect the best results for queries related to those subjects). ... When I'm not doing those things, I just use Google or DDG because they have better results for day-to-day queries."

Let me know if you have other questions!

[0] https://docs.google.com/forms/d/e/1FAIpQLSdf8lAoShQz7Wjl9h60...

[1] https://developers.google.com/custom-search/docs/overview

"It's a feature not a bug" (TM)

The site uses a whitelist of URLs to (attempt to) keep results relevant to science and programming. In the context in which I'm using this search engine, I have no interest in (reviews on) 3 hole punches. (That's not to say I never do, but in that case I'd use Google, Reddit, etc.) The fact that results don't show up here means that they also won't show up when I'm not looking for them, which is 100% of the time when I'm using this search engine. That's a plus for me personally.

Best case would be to have relevant results in a single search engine, but that's not what I intended when building this site.

From OP, "'No Trash Search' is very focused on STEM and not 'for daily use'". I would assume your 3 hole punch review search query falls outside of what this specializes in.

First test search "python random" - returned just what I would expect instead of multitude of low-quality blogs like Google Search does. +1 from me!

No, documentation about how to generate pseudo or secure random numbers in python. Not sure I got your question.

This is the approach I imagine Apple would take if they were to ever launch a Search Engine. A large corpus of handpicked sites.

Haven't tried Neeva but I can vouch for Kagi.

Also search.marginalia.nu puts a smile on my face almost every time I use it :-)

(I should try Neeva, I keep hearing good things about it.)

Pretty nifty project. I'm curious what whitelist are you using, would it be possible to list allowed domains somewhere publicly.

One suggestion that I have is to remove w3schools.com from the whitelist. MDN is a much better source for information about web development.

I agree with this approach. You can't crawl everything from the getgo - focus on a very specific domain catered to a specific set of users.

The first two results for “Rust awesome” were two ads which took up my entire screen on mobile. Both travel related.

The GitHub repo was third and had to be scrolled to.

Seems pretty trash to me.

Great job! How are the ads implemented and do they cover the costs? I'm thinking of building a similar search engine for a completely different domain. I am a bit concerned about paying for it though.

Thanks!

Ads are added automatically by Google. The whole thing is little more than a wrapper around the 'Programmable Search Element Control API' which is an HTML element you can just insert into any site, like an iframe. Unfortunately this is the only way to make Programmable Search available at scale as the API is restricted to either 10 sites or 10K queries / day, even when paid!

There is a paid version for the HTML plugin, but that would leak the API key and so it wouldn't work as a business.

There is an option to get a share of the revenue generated by a search engine. Maybe it's time for me to figure out how that works.

I was thinking of making a hosted, ad free, customizable version where people upload their own keys. Not sure if people would like that.

As a side-note, it's super easy to remove ads with 1 line of CSS, but I wasn't sure how Google would feel about that so it's not in the online version. TamperMonkey is an extension that allows people to insert their own CSS on different websites. Hmm.

You can view all offerings in the docs [0].

[0] https://developers.google.com/custom-search/docs/overview#su...

It would be cool if I could take my existing browser history, aggregate by domain, sort by frequency & then create the necessary xml for the programmable search. Maybe with a pick & choose UI so I could decide which sites I wanted.

Right now, looking at your allow-list config, it feels a bit custom to you, but if I had an easy way to limit search to the sites I myself know and trust, I could see how that would be useful.

I know I could probably pick it out of my browser's history UI & poke it into Google's Programmable Search UI, but that seems like a hassle and a half.

Good idea! Would you pay for a premium version where you can customize the whitelist (and additional features) if it were available? Bing charges around $5 per 1000 searches, so I guess it would cost about the same. (Google's API is limited to 10K searches / day, even when paid)

I think just the history->xml conversion would be a bit of a hard sell, but I would be willing to pay for a sub if there were some additional features like:

* easy add to my filter list (like maybe a browser plugin so I can see that the current site isn't in my filter, but I can click a button and now it's in my filter & opposite for remove for when sites start to suck)

* stats on which sites I visit after searching

* aggregate bing+google filtered searches

* curated site lists for different topics, top 100, etc. Maybe like a temporary search using these sets such that I can try them without affecting my own filters. Maybe sharing lists w friends

* some sort of search anonymization/log deletion feature

* integration with browser search on desktop & mobile

* search flags like duckduckgo so I can easily switch filter sets by typing like /news or /nerdshit in the query

* integration with archive.ph & wayback machine

That's a good idea! Not OP, but I'm creating a faster search engine for programming queries, depending on the tech searched it will also point to curated sites that could have the answer. Will try to implement your idea as well. Thanks!

How about using the Bing API? Isn't that more open?

With caching, I think you might be able to reduce the load.

Also, why is w3fools in the list? It's an awful site.

Yes! Maybe I'll build a premium customizable version out of it. Do you think that'd be useful? Bing charges around $5 per 1000 searches, so the premium version would be around the same (with caching covering hosting, maybe).

Googled "Best smartphone 2021" which resulted in crappy result. Maybe I am missing what significant categories actually means?

I agree those results are not helpful.

The poster did say it was mostly for STEM subjects though...

More importantly though, I think "Best smartphone 2021" is really a search that has been conditioned on the crap google gives back now. At best you might expect to find a "best smartphones" listicle or something.

This is just a whitelisted search, so in my 5 min playing with it, it looks like popular or consumer queries are more likely to just provide reddit or wikipedia links, while more technical searches land on SO or documentation sites.

I think with a little tuning, this approach is great. Given the modern internet and all the crap there is, a manual whitelist of sites that are actually legit is always going to be superior to an algorithmic approach.

How would you define a good result for that query? It seems like the only thing you could expect from that query is a page of advertising, since the "best X" without any criteria is a concept that doesn't exist in reality.

Somewhat true. A perfect awnser for such a query would be a page like tomshardware where the best smartphones for multiple categories would have been nominated. A normal Google search actually does return that, after some scrolling. This however is ofc heavily based on my browsing history.

(obviously: this is subjective, so what's significant to me may not be to you.)

Honestly I just created this search engine for myself to find things more easily while programming or studying (I study biology, cs and ai; and philosophy in my free time, so expect results the best results for queries related to those subjects). I think those subjects also appeal to the HN audience, that's why I shared it here. When I'm not doing those things, I just use Google or DDG because they have better results for day-to-day queries.

That being said, I'm definitely interested in helping improve other people's search as well (reason I'm posting at all), so let me know if you have suggestions for sites to add!

Thanks for sharing. My initial comment sounded a bit harsh. I am sorry for that. I will look into fine tuning it for my needs. It is an interesting approach to a very annoying problem.

NP :) It's exciting to have people using & commenting on something I build.

> I will look into fine tuning it for my needs. It is an interesting approach to a very annoying problem.

If a premium version were available with a customizable whitelist, would you pay for that? The API is around $5 / 1000 searches so it would cost about the same.

There is none to my knowledge. That is why I tried it. Google returns at least links from tomshardware, wired and Android Authority after a few scrolls.

The point I was attempting to make is that what you are looking for doesn't exist within the realm of authentic websites made by people. The only ones making content to match such a query are deliberately attempting to match it because there is so little quality content. So what you get is at best the sort of useless content mills that seem to crop up on Google.

I did a search for "eternal crusade".

The blob of the ads are still the top results. This is not the "no trash search" I'm looking for.

the github repo is just few html pages i dont see anh source code.. i wont trust this search engine...

Glad you're interested in the source code!

As explained in my other comment, this website is a wrapper around google programmable search. The actual searching happens on Google servers, and I can see why people have problems with that. The code you see on the website is the same as the repo, though. It's actually hosted by GitHub! You can verify this by opening the web inspector in any browser or looking at the `.github.io` portion of the URL.

You can learn more about Programmable Search here: https://developers.google.com/custom-search/docs/overview. NoTrashSearch uses the 'Programmable Search Element Control API', which is documented here: https://developers.google.com/custom-search/docs/element and can be used with very little code!

I think your site is great, I've thought about something like this before but didn't realize how simply it could be implemented.

Stupid question though: where is the list of whitelisted sites? Is that something you set up separately with google? I scanned though the code and expected to find a list somewhere, but obviously you do it in a different way

Yes thank you, this is helpful! I think the list of sites that are included is the most important information you can provide about the project

Show HN: No Trash Search

news.ycombinator.com | 19 days ago

Paul Graham's Twitter thread on Search engines and SEO spam | Hacker News

The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence. It's a ridiculously hard problem, especially when people are targeting you directly. But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make the problem worse.

I'm not saying that curated search results for particular verticals is a terrible idea (though I'm sure like anything the devil is in the details), but on the whole Google search is very, very good considering the constant assault they are under from spammers (which most other search engines are not, at least directly).

The problem isn't that Google doesn't employ these people or invest in their activities.

It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities.

If Google:

- Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)

- Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool).

- Allowed you to create your own custom search engines, like "Programmable Search Engine".

That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.

Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.

Your point is good but I'm not sure I'd say very good given how easily the same SEO spam domains can stay at the top of search results for ages simply by scraping someone else's content. What I'd be most interested in knowing is what their success metrics are defined as — for example, how much of a problem does Google's management consider it if someone searches, finds the answer they were looking for on someone's Stack Overflow rip-off, and stops searching? I could easily believe that a significant amount of what we're seeing here is that they're focused on some kind of user frustration metric which doesn't include things like damage to other businesses.

Yes, I've noticed this particularly with technical results. A lot of sites seem to have scraped StackOverflow and GitHub issues, put a crappy ad-loaded interface around them, and somehow out-rank the original SO/GitHub content.

It's like the bad-old-days of ExpertsExchange, which somehow was never delisted by Google for its shady SEO tactics.

You just have to look at Google's profit motive here. Their motive isn't to provide quality search results. Their motive is to show users ads, either in the search results themselves or on the destination sites via their ad network. The SEO spam sites aren't a bug, they are a feature of Google's profit algorithm. Google's search quality will never improve so long as their motivation is to show you ads. Why should it? Competition may help here, either by an outsider like the OP suggests, or via breaking Google up with anti-trust enforcement, or both (my preference).

As a user, your best personal and ethical move is to install an ad-blocker, to make ad-based business models less viable, which will help promote business models that don't abuse the customer.

The core problem, I guess, is that search engines view all their results as ads. That’s why they got into the ad business in the first place.

This 100%. In travel, we see Google constantly tweaking its algorithms, and compared to Bing, Google surfaces a ton more small, well-written travel blogs [1]

Not only that, Paul and Michael have seen plenty of startups, and at least in recent memory, the number of vertical search and consumer startups that Y Combinator has funded hasn't been that high

As a consumer startup, I know this issue firsthand. Paul and Michael assume that if you build a better product, they will come! That's simply not true these days.

Instead, you need to:

- Build a better product

- Option 1: Figure out a channel with enough growth on an existing platform. This likely means you're doing SEO for your new search engine

- Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads

As the founder of Wanderlog (YC W19; https://wanderlog.com), a consumer vacation planning app [1], I definitely remember the idealistic days when I thought the best consumer product on its own would win! But growth doesn't just come, and the same can be said of vertical-specific search engines.

[1] Try searching "[your city] itinerary" on Google vs. Bing: it's much more likely you'll find a small blog rather than Lonely Planet or the local travel bureau as the top result

Hi! I used Wanderlog to plan a recent month-long group trip, which was definitely the most complex vacation I've had to plan. For context I am very active when traveling (e.g. multiple activities each day); so not sure how my experiences map to others.

The best part of it was (going to a foreign country) being able to find / identify all the attractions relative to each other, so I could go to cluster A on Monday, cluster B, on Tuesday, etc.

The hardest part of it (and why I needed to create a separate google sheets anyways) was--once I figured out opening hours of different locations, hard-to-book activities with limited reservations--the ease of moving things around more fluidly e.g. cluster B on Monday, cluster A on Tuesday, etc. and having a more information-dense view so I could see larger portions of the itinerary at once.

It would be cool to have an "input everything" --> "input time restrictions / unmovable things" --> output planned activity cluster type workflow.

>> - Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads

nonoonononooonono. No. Don't monetize anything for the first 10 years. That's the only way it can work. Then you can go monetize it and buy an island and not give a shit if you destroy what you created.

Oh but don't worry. You'll have investors.

Also, i'd be very surprised if they didn't have tens of thousands of workers aiding in spam review already.

The hard part in all of this isn't finding and stopping spam - it's defining what spam is. Are all the pie recipes where there's a 2000 word essay about their grandma at the top 'spam'? They still have the recipe, and Google Home devices pick up the recipe instructions just fine so people end up not reading it, but many people would still consider that spam since it adds such an obstacle to getting the information you want. Same for cnet articles like "Best smart home devices to buy in 2022" - it's a reputable brand with a list of smart home devices, but it's hardly a review and exists to funnel people to their Amazon affiliate link.

> The hard part in all of this isn't finding and stopping spam - it's defining what spam is.

This is one area where Google could use personalised results to provide a better experience for the user. Let me decide what spam is for me. Let me mark results as good or bad, so that the algorithm knows what kind of pages should be prioritised or filtered out the next time. Google SearchWiki was a step towards this but they killed it off.

Let's have niches where the content is hand curated by human beings instead of pure statistics by machines.

Hmm why stop there let's actually make the users do the curating and even the content creation by rewarding them with social validation. Let’s have hard working moderators who work on the community full time.

Then we could just build a search engine over it. We could call it Reddit. Or HackerNews.

Maybe the users aren't all as good as professionals at curating the information. Let's hire professionally trained curators pay them well and we could call them newspapers. Then we can come in disrupt them and replace them with an algorithmic marketplace that eventually becomes infested with click bait.

> and Google Home devices pick up the recipe instructions just fine so people end up not reading it

I think this isn't entirely related, but that's perhaps the beginning of a bias you might end up having that everyone experiences technology in the same way as it marches on. I've yet to encounter a Google Home in the wild, I imagine far more people are consuming recipes on phones, tablets and PCs.

AFAIK the 2000 word essays in recipes are Google's fault - it prioritizes pages with a lot of content, so you have to add that junk to the top in order to rank highly. While I'm sure there's more going on behind the scenes than I'm aware of, it does seem like the rules could be altered on a category-specific basis where a lot of text isn't necessarily a positive.

Years ago I wanted to pursue micro blogging, but this "feature" of Google search stopped me from doing it.

What's the point of writing succinct, to-the-point mini articles about problems and solutions if nobody finds them on Google?

It's two-fold. If Google priortizes pages with a lot of content that's one thing, but longer content also means more space for ads, or more scroll events to trigger ads, etc.

Incidentally, prioritizing long content seems odd to me, in my experience the best pages are short and get right to the point, at least in the context of something like a recipe or other "how to" resources.

Yeah, the newest nuisance seems to be sites that clone Github Issues and StackOverflow with a crapper interface. Somehow they rank higher than the original sources. I'd say it's spam but it's definitely not traditional spam.

And the strange Wikipedia mirrors that are shown in Google Verbatim searches instead of the original. If I disable Verbatim, they disappear and I get regular Wikipedia instead.

The curated search results business model doesn't work. Google gives "aggregators" and other search engines the death sentence for organic search traffic from economically meaningful queries, so you'd get no free traffic. This is one of the major antitrust complaints against Google in the EU. Since you get no organic search traffic, you need to build a brand using advertising, and once you start down that road you need to monetize the first click which compromises the quality of your site.

I said this in a similar thread yesterday, but I think this is an unsolvable problem because much of the content either no longer exists in website form or is old.

To put it simply, a new generation of the people who used to make the reliable niche websites that not just answered your questions but also helped you learn a particular topic have moved to youtube instead.

Google search is hollowing out as a result with the meat going and the SEO'd fluff that kinda answers the question but ONLY the direct question being asked with none of the wider expertise that more educated people in what they were searching for.

Of course google owns youtube as well.. so perhaps they just see it as an inevitable transition.

Just a note on that, youtube search is finally getting better, yesterday I noticed it was able to find key words in the middle of a lecture that had nothing in the title or comments. I always wonder about their AI transcription service, it's gotten so good, if they're storing all that audio as text, I guess their search is going to get excellent?

This x100000.

There is no scenario - none - where thousands of engineers at Google working on search wake up in the morning and say "we sure have made it good enough wr2 SPAM. I think I'll have another Danish."

I agree with this and the grandparent comment wholeheartedly. That said, there's a kind of institutional blindness that can build up in companies—especially ones that dominate their sector. It may have roots in intransigent upper management, ossified and inflexible process, wide-scale burnout, a culture of passing the buck, or any number of other pathologies.

I don't claim that Google has any of these and certainly have no insight into their search group. But I've personally been at powerful companies with best-of-the-best talent that were blind to the decay in their own living room, so I would caution against immediate dismissal of PG's take.

When the cafes were open, you can bet they said, "I'll have another Danish, and then get back to work on this problem that never seems to go away."

I sure wish I had problems that were totally unsolvable, they are so easy to measure progress on. /sarcasm

I think it’s more likely that because they are just building hundreds of tiny tweak experiments and it’s someone else who desides what to build and if it even worked. Search quality is such a meta-problem that it goes beyond any real hope of simply working on it in anything beyond piecemeal trial and error fashion on their dataset.

> I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence

Really?

I can point you to Hard Problems that have been solved better at little startups than at Google - or, indeed, at any other bigco. That's why acquisitions happen.

Why does Google having 1000 engineers working on a problem automatically mean they are the smartest?

Really? I don't think the bureaucratic bloat at Google cares and the original authors of the search engine in its current incarnation are probably long gone. It is maintenance mode and I don't think they dare touch too much.

It takes time and effort to build up a spam site's ranking but it is trivial to blacklist those who get to the top.

> But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make the problem worse.

The reality is more that some Google engineer will come up with an algorithm change that makes the result 40% better, but it will come at the expense of making that search 3ms slower so the change won't get merged. Or it will make the results worse for some niche set of queries that the business team really cares about, so again it won't get merged.

There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results, so there is lots of room for a startup to compete.

> There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results

Google has a paid-for Search API, so they could do that if they chose to pursue it. And then they could let Google One users opt-in to the same thing via ordinary Search. I'm not sure whether Bing has anything equivalent.

> It's a ridiculously hard problem, especially when people are targeting you directly.

In general you are right.

But there might be a sweet spot somewhere between solving general AI and allowing Mahalo[1] and similar sites to pollute search results for months.

Well of course back then this was the problem: auto generated content that matched queries and ranked highly.

Today Google has created a bigger problem of their own: not caring what I search for at all.

Mahalo could have been removed with a (user configurable) block list.

[1]: https://en.m.wikipedia.org/wiki/Mahalo.com

I understood PG's point differently. My understanding is that he is suggesting an angle of attack in which carefully crafted manual reviews (that do dot scale) can be used to bootstrap a product that does scale thanks to something else (e.g. collaborative filtering). All of this being on a niche domain where you can drive a wedge into the mediocre performance of Google (online shopping probably being the worse possible choice, but there are many others).

I agree. You can say a lot of bad things about Google, but they definitely have some of the smartest and highest paid engineers working on their search. Plus there are already a lot of people trying to compete with Google and so far, no one seems to provide consistently better results.

The only advantage a startup might have is that they could do completely new concepts, such as specifying what area you search in, allow you to modify their classification of your query and/or moderating sites you include - which is probably necessary anyway, since you'll hardly have the budget to fully index the web. I'm not saying it's impossible, but it's not going to be easy at all.

And after all of that, you still need a way to make some money.

Most comments focus on the technical side of things, whereas I'm sure there are also legal restrictions involved in this. If Google delists a website on the grounds that it's a copycat of stack overflow, or because they have low quality content according to Google's taste, there might be lawsuits filed against Google, claiming that the company is discriminating.

How hard is spam, really. If you're Google? Here's what I would do as a heuristic (uh, not evil?): We know everything about you and everywhere you've visited and everyone you've talked to in the last 60 days. We know all their phone numbers and email addresses. We even know the girl's phone number you met at the bar, who didn't give you her phone number. So if any of those people email you, we'll categorize that as "not spam". Also, if it's your boss or a coworker, "not spam". If it's a major company that's existed for more than ten years, not spam. Everyone else, spam. Done.

This is hyperbolic, right? But they can solve spam in a split second, if they just admit they're watching you all the time.

To some extent, I worry that the problem with search engines is that there isn't any data worth returning. Yesterday's thread talked a lot about reviews. Writing a review is hard work that requires deep domain expertise, experience with similar products, and months of testing. If you want a review for something that came out today, there is no way that work could have been done, so there simply isn't anything to find. Instead you'll get a list of "Best TVs 2021" or whatever, with some blurb and an affiliate link, not an actual review. That's what people can make for free with a day's notice, so if you write a search engine that discards those sites, that's fine, you'll just return "no results" for every interesting query.

I guess what I'm saying is that if you want better reviews, you probably want to start writing reviews and figuring out how to sell them for money. Many have tried, few have succeeded. But there probably isn't some Javascript that will fix this problem.

> If you want a review for something that came out today, there is no way that work could have been done, so there simply isn't anything to find.

That's not strictly true, given that reviewers are often sent pre-release versions of things in order to do that work before release day.

>This may not just be a problem with Google but possibly also the recipe for beating Google. A startup usually has to start with a niche market. Why not try writing a search engine specifically for some category dominated by SEO spam?

>You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant).

Is he describing...Yahoo circa 1994? A manually curated directory service.

Some of the examples used in the Twitter thread Paul was referring to would be better served by a manually curated directory service with a possible addition of a search engine only surfacing content from the sites in the directory.

For health information and recipes in particular there are only a handful of really high quality sites that have quality content for 95% of the information most people need. I bet if you wanted to increase the coverage to 99%, that list would expand to less than a thousand sites. At those numbers manually curating the information would be easily achievable.

How to get people to use your top notch Google replacement instead of Google, however. That's the hard problem.

I wouldn't just complain about Google. Google search results mostly reflect a deeper problem with the web today. I do miss the simplicity of the 2000s.

And makes me think that StumbleUpon had a similar curation ability, in that the value qualifier is how often [hopefully] real people interact with content - tracked by who's using SU and agreed to allow tracking; can't remember if sharing that was optional or not?

The gamification of the system then would have to come through onboarding fake users, pretending/mimicking real user behaviour to send that signal into the system; not sure if SU ever ran into that problem or was actively paying attention to trying to identify and removing fake or suspicious signals from their output?

I feel a much better system is easily within reach, it's simply getting the right structure to it, the right foundation, and then it will quickly take off due to the quality difference. I've already figured out a design pattern that Twitter and Facebook has indoctrinated us with, making us think it is normal - and keeping us blind to an actual normal way or organizing or communicating, but that isn't conducive to control or ad revenues - and so extending my future plans to include a better search-directory system would fit snugly into my efforts.

    Why not try writing a search engine specifically
    for some category dominated by SEO spam?

I like to compare search engine results and wrote this tool to make it easy:

https://www.gnod.com/search

There in fact are many vertical search engines. You can click on "more engines" to see the whole list.

Okay that's sort of what the !bang in DDG is for, and why it's a meta-engine. What's the blue sky ideal for a real, no-bullshit, everything search engine that doesn't fall prey to the constant flood of garbage?

I have an exterminator who comes to my house every couple months, and sets up traps here, poison there. I don't have any rats in my house. I do see rats running across the yard sometimes. The exterminator explains it like this: Rat pressure. The rats overpopulate and there's "pressure" (like, uh, "memory pressure", which is also a fluid concept) so they try to get into your house more, through smaller holes, as a function of how much outside drama is going on, how many they are and how overpopulated, how scarce their food supply is, how cold it is outside, and whatever else drives rats into your house. (I love the dude who's my exterminator).

Anyway, this is the same problem every search engine faces. The more surface area they expose, the more pressure they have building, the more ways people have to fake out their systems.

We have to go back to the 1990s Yahoo! model. Curated content. A list of websites that are reputable. 1990s Yahoo is the future.

Creating a "trustless" search crawler, where anybody can participate, and then applying an algorithm to determine trust or value feels like it'd be a never-ending arms race - that'd require AI and extensive/expensive resources that is likely better invested in developing real trust networks and curation; curators are corruptible and regulatory capture of policy is possible if the organization is infiltrated or poorly overseen.

Carte blanche opening your system up to anyone to inject data seems like the wrong foot to start off on, whereas my curating a moderator, someone I personally know and feel good about, trust to whatever level, and hiring them - ideally making sure they're someone you respect and you're someone they respect, pay them well, and at scale will be able to pay for itself; this did just bring to mind however big pharma and pharmaceutical trials structure and how that system can be/is/has been captured - and so perhaps the pressures when dealing with multi-billion dollar market categories will always lead to shenanigans if ever trying to centralize too much, not allowing for de-risking and broader resource distribution via sales/profits to more parties than the "5-star" rated products.

In a thread on HN, I think it was yesterday, a few people posted about review sites where some product reviews are free - but others you had to pay for. A system to facilitate such organizations could allow a highly competitive environment, where organizations develop/build a brand - build trust for their brand as being competent and thorough - and so then over the lifetime of a reader/customer, perhaps they'll spend $1,000 buying reviews (say 333 big purchases over 40 years that you're willing to pay $3 a hit for) to make sure they're ; mind you there will be organized that could be captured to say promote one conglomerate of products over another, perhaps even regionally, but I'm beginning to think it's a necessary layer to combat the shit show that is Amazon (et al) reviews. Ideally these systems and how the reviews present the information, and how thorough - the technical depth and breadth and testing done - will help educate those who dive into using this system, which will sharpen themselves while keeping reviewers on their toes and arguably strengthening their organizations and competency as well.

The fact that you included Reddit, SO, Google Scholar etc is awesome (I thought it was only for main search engines). Thanks for sharing. Bookmarked.

This is already happening for a bunch of verticals:

Travel - Expedia, Hotels.com, Kayak, etc.

Consumer Goods - Amazon, WalMart, EBay, Etsy, etc.

Automobile Purchase - Cars.com, Autotrader, etc.

Career/Job - Indeed, LinkedIn, etc.

As Google continues to lose search volume on these big revenue categories it is going to make spam much more difficult as they are working to sort out long tail spam. Way harder.

Wow so that's completely opposite of my feelings on this topic. I would never, ever use Expedia for travel search over Google flights/hotels. Google travel is the meta-search engine for this vertical. Expedia etc. are all-in on spam and scams, trying at every opportunity to take an extra dollar from you.

The same with Amazon. You'd think that with all the armchair search quality experts cropping up lately there might be more vocal complaints about the fact that Amazon's own search can't find basic consumer products sold by Amazon itself. If I want to find stuff on Amazon, I search Google for it.

Just give users the ability to blacklist domains when searching; pretty soon you'll have a decent list of what users consider worthless.

And pintrest would die.

I think a lot of this is due to Google both owning search and the ads on the websites (AdSense). There’s an incentive for them to prioritize click farms (and other sites filled with their ads). I think in general there may be a correlation between the number of ads on a site and its usefulness to me, which is inverse to its usefulness to google.

I’m curious what would happen if those products were split up into 2 separate companies.

I also can't help but wonder this. I'm certain people at google search want to provide the best quality search results and do this with integrity. But at some point in the business hierarchy you are at a level where people set objectives for both these departments ( search & ads ) and are trying to optimise for things like total revenue/profit.

Yes. In fact if you wanted to cut out the SEO spam then delisting anything with Adsense would probably be a good start for a competitor.

Good idea. You could start with fitness. Lots of high-quality information out there that’s entirely, 100% inaccessible via google.

Over COVID, I did the whole fitness thing from a few different angles (overhauled diet, trained for a marathon, now lifting weights a lot). I found I could only find good info by going directly to a trusted source - literally, typing http://www. like I’m in the 90s or something. This is the exact issue a search engine should solve, but Google doesn’t.

This is a problem I'm working on.

What sorts of unique things did you do that Google failed at? Maybe you read through discussion sites or got tips from books or something like that

Is your trusted source the same as my trusted source which is the same as my neighbor’s trusted source which is the same as my Australian cousin’s trusted source?

If not, at least one of us is going to hate this new search engine.

> Why not try writing a search engine specifically for some category dominated by SEO spam?

Back in the olden days, there were lots of organizations that collated high quality content from the best writers. They nurtured expert writers and paid them well, fact-checked the content, and employed diligent editors and proofreaders to ensure it was as accurate and well-written as possible. Over the years, they'd build a reputation for reliability and trustworthiness that kept people coming back for more. If you wanted to learn about fitness, or cars, or cooking, or science, you'd find a reputable author and publisher and buy their magazines or books.

But then in the early 2000s the geniuses from SV "disrupted" the publishing industry and its financial mode and brought us a much better way to find content, the search engine. Because they were so much better than the old-fashioned publishers, search engines gobbled up the advertising money and became the dominant gateway to content. Publishers were forced to abandon expensive high-quality writing because rankings and eyeballs now mattered more than quality and trustworthiness. Instead of investing in writers, they invested in marketers and SEO specialists.

End result: crappy content, writers banging out garbage for peanuts, and useless search engines. Two decades later, looking at the barren wasteland they had created, the SV geniuses thought: I know what we need, more search engines, but smaller ones that collate high-quality content from the best writers. There must be money in that, right?

I have a version of fixing this that I would personally enjoy a lot. Leave google alone, let it crawl the web, prioritize what it wants to via algorithms. But, give me a version of that which ONLY surfaces results from discussion forums (including SO, Reddit, HN, etc). For most of the stuff where I am actively searching and not just looking stuff up, discussion forums of motivated, self-selected contributors have the stuff I need with the context I need. It used to be that blogs had answers, but that media has been categorically ruined by SEO.

Now, one of the deficiencies here has been examples. Try this: "best miter saw". you will not find any websites that actually discuss the answer to this question, despite it being a product category with a lot of price variability and performance tradeoffs (weight, capacity, power, cord vs cordless, accuracy).

Nearly any product reviews for large purchases follow the same pattern unless consumer reports has decided to dig deep (e.g. washing machines).

How about guitar strings? Sandpaper? Printers? google's algorithm has allowed profit motivated websites to displace the commons to too great an extent.

You may be interested in searX - it refers to data sources as Engines; you have the ability to run your own instance (or use a public shared one) and only enable engines you want results from (reddit, stackoverflow etc.). Build your own meta-engine recipe, basically.

https://searx.space to learn / get started. Find one and visit it, click Preferences upper right then follow your schnoz.

https://search.marginalia.nu has a very interesting approach to this:

Use tracking especially and JS generally as a weight in ranking so sites that contains much of any of these needs to be exceptionally high quality to float to the top.

This means sites with limited ads and tracking, typically enthusiast driven pages float to the top.

Now always when someone discusses a novel way to combat webspam someone will immediately counter: if this becomes popular SEO hackers will immediately start doing this.

Well - if reducing page size and removing tracking becomes a leading SEO trick I can deal with a bit of SEO hacking :-)

Yet for some reason I feel Google won't start using this very simple metric :-)

Another big factor in what I do is prioritizing the opinions of certain indieweb sites when ranking domains, basically a segment of the graph consisting of humans with a particular dislike for seo spam. This makes ranking manipulation much less effective.

How can they tell it’s a discussion forum? Does the scraped search spam that copies stack overflow content look enough like a discussion it fools their heuristics? Is it a manual process (in which case you can bet it “doesn’t scale” and won’t be built) this is the problem they face. Literally nothing is simple given the size of their dataset, the scope of their user base, and the adversarial nature of the very world around them generating new data they must work with in order to do the job.

There’s definitely an element of “we got our profit so fuck it” with respect to the search engine advertising business and Google’s incentives to make search quality better, but that doesn’t change the difficulty of the underlying problem. If I wanted to pay Google $5 per search for super high quality results, even $50, they can’t just make this product better to get my money, they are fighting an ongoing war against adversarial SEO which prevents this from ever being better than stalemate at best or more likely due to economics, the slow slide into declining quality we see due to the SEO side having more money with which to pay for engineering brainpower.

My current solution for this is to just tag `site:reddit.com` to the beginning of Google searches. A Google search for `site:reddit.com best miter saw` has a lot of relevant results.

Marketers/SEO people are starting to infiltrate this as well, but since they can't control and SEO the content on Reddit nearly as much, this still works pretty well for now.

Of course! this is a tip I got from HN years ago. it gets old to add Reddit, PracticalMachinist, Fine Woodworking, etc. I really want a proxy for user generated content where nobody got paid to write it.

Fun story: I knew a person who worked for a home building website part time. She got paid to write stories on home renovations. She had never done _anything_ she wrote about. Mostly, she gathered up other blogspam and recycled and rewrote it without citation. Sometimes she went to forums, sometimes reddit, sometimes youtube. But, the universal part of it was that she had to produce 2x pieces of content per week endlessly. Just for a local LA builder. Most of the content wasn't "wrong" but, it also wasn't exactly incisive and didn't include any details that would have been useful. Instead it was just filler. The worst part is that it consistently improved that company's search ranking.

Content farming needs to die.

Interestingly, Google Maps doesn't suffer as much from the issues with Google Search. Maybe because it has those community-driven curation features that PG is talking about? Google Maps is fantastic at finding places to go to (and getting you there).

Also, why hasn't Apple built a search engine yet? It baffles me that they chose to go head-to-head with Google on Maps, yet outsourced their search engine. I would've liked it the other way around: Google Maps and Apple Search.

It’s much harder to get SEO spam style content to maps given the geographic region limits involved… But it happens my favourite example is searching for suppliers of something basic, say structural aluminium extrusions, big and heavy and you ideally don’t want to ship it far so it’s an ideal thing to search for a local supplier of. In Australian results is basically a given that I’ll get results for my city because they will list it as a delivery area they supply to however when you actually try to find them on the map as a pin, nope it’s either not their or just is a sales office not a warehouse or workshop, so they have tricked the system into listing them as local to my area in a way that pollutes my maps search results.

But this only works for certain industries. It’s much less common to see this kind of tactic if your searching for say a coffee shop because they sheer number of local results let’s Google be “hyper local” with these kinds of results.

Apple makes 10+ billion a year from Google by setting It to the default search engine on iPhone.

I agree they should make their own search engine. But currently they’re being paid a ton of money not to.

ah, so everyone wanted to move from carefully crafted personal websites where every detail counts and low effort publications are harshly punished to platforms with guranteed readership and now we have a curration problem?

Someone (who probably doesnt have a website) said that comment moderation on your own website is to much work. Perhaps the whole internet is to much work?

But i like the spam search engine by and for spammers as a way of finding the latest and greatest affiliate marketing and blockchain swindle.

He’s just describing Webrings, except in a reactive tense (“filter out spam sites”) rather than a proactive tense (“associate your site with other worthwhile sites”). Google’s ranking algorithm only works when someone is proactively curating, and only SEO spammers do so these days. Reactive curation is not a viable way to manage information.

The simplest way to compete with Google is to create a DIY Webrings site that disallows harvesting of data by Google. Charge curators to create a webring, and let curators select three hashtags and a description that represent their list of fifty or fewer sites. Use the revenue to pay a human to curate the list of hashtags, and let users tip a webring curator in gratitude with an Apple Pay button.

This is how to make a million dollars, Pinboard-style, out of the ashes of the original curated Yahoo idea and the information structures of hashtagging. It doesn’t work if you allow free-for-all infinite-sized lists, it doesn’t work if you allow free-for-all hashtags, but with clear limits and moderation of tags (instead of webrings), it would thrive. By moderating tags, users can keep the webring they paid for, and SEO rings will be stick out for having no shared network with any other rings, which allows for easier detection and culling of malicious non-participatory actors. Plus, with the curation networks in place, it becomes possible to bubble up rings that have unusual content for positive human moderation activity.

I tried to find some good podcast lists yesterday and each site I visited had a really interesting cross-section, but there were so many duplicates. I wish the ring site existed, so that it could remember what it had shown me already, and I could say “show me rings that intersect with this podcast and have something new I haven’t seen before”.

That’s where the theory of pagerank and the practice of curation and the capabilities of search align, and given that moderation of hashtags scales very cheaply, is a billion dollar opportunity that Google and Amazon cannot compete with if handled properly. It’s not about trying to get a cut of every visit’s revenue potential. It’s about giving human beings a directory that respects their time and remembers what they’ve seen.

If I remember correctly, it used to be called About.com - with categorized and human curated links.

It was big during the dot com days, but withered after Google.

Interestingly, I do think that that model may need to be revisited.

I believe that the only moat protecting $100B of AdWords revenue is the quality of the Google Search results. There is no meaningful switching cost to using a new search engine, and the spend inertia in ad spend is not very significant (i.e. any online marketing manager will happily spend 5% of their budget in a different search engine adwords-like program if they get better ROI, there is no incentive the be a "Google Ads only shop").

On the other hand, Google needs to maintain the ballistic trajectory of its revenue growth. So how can they fix search quality when they've minmax'd themselves into this situation in the first place? If they were to make the ads background yellow again, that would have negative short term effects that I doubt any career exec can stomach.

“ There is no meaningful switching cost to using a new search engine.”

Unfortunately, defaults matter and Google is spending billions of dollars yearly to make sure they are the default search engine wherever they can. Most people don’t switch from default.

No.

The other moats are lock-in to advertising networks, website metrics, and effective control over Web standards through the Chrome browser.

An alternative search platform might provide better search. It would be fighting Google on at least three other fronts. It might have some success, but it would be challenging. (As history largely demonstrates.)

Even a rival tech monopolist, Microsoft, barely holds even with its own search offering (I use that indirectly via DDG), and scrapped its own web-browser development

Just a minute ago, I made a small typo in a non-obscure programming-related search term.

    Showing results for searchterm
    No results found for searchterm

Followed by an unending list of random celebrities I don't know nor care about, businesses I've never been that sell items I have absolutely no use of, and random foreign news articles.

Failure to recognise the typo is unexpected but forgivable. But then, rather than helping me with my search, they attempt to distract and try to lead me away from my task - using triggers that you'd think they should have known wouldn't work.

I really don't understand how this is even possible, and it's not a rare occurrence.

It'd be really interesting if Google allowed upvote/downvote on search results... but it'd be super hard to imagine them every taking the votes into account much versus ad revenue.

And the upvote/downvote would be very tricky to implement in a way that the SEO crowd couldn't just game it horribly.

I'm not very well versed in SEO but isn't this just good old Goodhart's Law?

Come up with criteria to determine which websites are "better quality". Measure them, rank them, put the ones that fit the criteria best at the top.

On the other side, there's the people promoting their websites. Do what you can to get as close to Google's ideal as possible through whatever means. Profit.

At this point the criteria becomes useless for any real quality analysis.

Was this linked for the irony of everyone spamming replies advertising their startups which don't solve the problem but kinda-sorta do, resulting in something hard to read and understand?

> Maybe ultimately you open up spam fighting to your users. If you managed this well, you could harness a lot of energy.

Doesn't Google already consider that if a user returns to the results page (or clicks a second link) then the first link visited was not satisfactory. Seems like a pretty elegant solution.

What are some search categories that are so dominated by spam that they are unusable?

I'll start: "how to rent a car" [0]

[0] worth noting that I personally get somewhat reasonable results for this, with a 3rd result from nerdwallet.com and a 4th from wikihow.com, both of which seem to answer the question in an unbiased way

Nerdwallet and wikiHow are both SEO spam content farms. They just happen to have above-average quality content.

They don't exist without a search engine.

Could this issue be related to Gmail's spam filtering? For approximately 2 years now it's been downright porous, I'm getting on average 1 obvious spam message in my inbox that is something like:

c0nGrats-You_HaVe_Won_ThE_Pr1ze!

..Or some silly variation of this that takes literally 0.1 ms for a human to discern that it's spam. Yet something happened to Gmail's spam algorithm in the last couple years that has been consistently letting these through. To be fair, it does catch most spam but it's only batting something like 75% and the spam it does catch is often times much less obvious to human eyes than the stuff it lets through.

All of the amateur search quality experts forget to mention the regulatory environment. Obviously, Google could nuke Pinterest from orbit, dramatically improving image search results. Clearly, Google could effectively take down Statista, technically. But various Eurocracies have shown an extreme willingness to take the side of Yelp, Pinterest, and whatever other spam/scam mills are able to form a shadow alliance with Microsoft's astroturf campaigns like "fairsearch" and whatever.

Is there a search engine for programming? One that not only searches stackoverflow, github, relevant subreddits and the other big sites, but also finds programming articles in personal blogs?

That would be valuable to me.

> What would a paid version of Google Search results look like - where Google can just try to give me the best possible results and not be worried about generating revenue?

God please no. YouTube premium shows what Google would do, i.e., they would further ruin the free experience by ramping up the amount of ads you see to "incentivize" the premium search.

Premium offerings like that are amazing simply for the fact that you can 'return' to the days where you obtained services by paying for them directly, not by looking at ads and paying with your mindshare. Google and YT aren't free services, and it's a miracle they continue to be fine with the notion of keeping it all accessible with ad blockers enabled.

Searching code is also impossible on Google. If there’s a competing search engine for that I’ll use it at least for this use case.

Several people mention DuckDuckGo in that Twitter thread. I use DuckDuckGo for my main search engine, and it's not obviously any better than Google regarding SEO spam.

One example, the website called gitmemory which crawls github data regularly and have better SEO than github that usually you will find results above original github links.

One approach would be to have moderators from the community who are allowed to make decisions about results.

Typesense looks easy to use.

But with 3x memory needed for the indexes, the server costs probably aren't going to be bootstrap'able.

Especially for a "small" crawl of a billion web pages, event at just 10k per page.

We still need a search engine that actually blacklists everything serving ads. Google beat altavista, now we need to beat google.

I mean no mincing about- recipe sites that are ads are blocked. Results with pixel tracker etc are blocked. Hell, results that are paywalled are blocked because they're useless.

When the search engine is funded by ads, there is incentive to produce results that people who click ads like.

A lot of the spam results just seem to be copy pasted content.

I wonder how difficult it is to compare the main body of text in search results, then say if it is over a 95% match with another site (I.e. it has been copy-pasted), demote it in the search results. If a site generates too many of these demotions then it gets blacklisted from the index.

> And boy would Google find it hard to follow you down that road.

This is a good perspective. Where can Google not go? Places that don't lead to profit. They will try (cough Wave cough) but will give up.

> Lots of people want to be amateur police. (pg)

This is very true. How many times have I clicked on a site met with ads so bad that the browser slows down, and after 10 seconds the page gets covered up by more and more crap and then a paywall shows up sometimes too. Now here's the thing - a competitor to Google might detect you clicking back and then pop-up a special set of controls near the search result that lets you say: "too many ads" or "paywall".

However, if such an engine were to start beating Google, I'm sure Google would implement it in their own way: automatically detect why you clicked back in such a short timespan.

> However, if such an engine were to start beating Google, I'm sure Google would implement it in their own way: automatically detect why you clicked back in such a short timespan.

Do you seriously believe that Google doesn't use that as a datapoint already?

Sooner or later you’ll have to deal with ballot-stuffing - companies trying to bury their competitors by casting lots of negative votes.

Perhaps ML will help in detecting such campaigns.

I'm honestly not trying to take a potshot against PG or YC here, but it's kinda funny to see him saying this after I worked for a YC-backed startup years ago that built its core revenue streams around generating SEO spam, we just marketed it as something else. Just to be clear, I don't think PG or YC are responsible for all or even most SEO spam, but I know firsthand that they've profited from it through at least one of their incubated companies.

I never considered the possibility that an incubator would support a specific product, then later on call for alternatives that would essentially freeze out the original product that they supported. I'm sure this very rarely happens, but it's interesting to see a real-world example in action.

I don't think it's as counterintuitive as it sounds - just because they're playing the game doesn't mean that they think it's right. If your options are to not be successful at SEO or to do the SEO spam thing, I don't think it's necessarily wrong to do the latter - it's your job to make your startup successful, not to make a stand against the way Google does things.

I view it as something like the rich folks who call for additional taxation of the rich. They're not going to just pay extra money that they don't have to under the current tax rules, both because it's not particularly fair and because one person paying extra taxes, even if they're very wealthy, isn't going to make a big impact. That doesn't mean they can't lobby to change the rules and be totally fine with it if everyone is paying additional taxes.

Paul Graham's Twitter thread on Search engines and SEO spam

news.ycombinator.com | 19 days ago

A modest proposal to fix web search

The problem with search these days is that Google has begun to produce absolute garbage results.

To try it, just search for something simple like “css center element”. On a fresh session of Tor Browser, here’s my top 10 items on Google:

(https://www.w3schools.com) CSS Layout - Horizontal & Vertical Align - W3Schools
(https://www.freecodecamp.org) How to Center Anything with CSS - Align a Div, Text, and More
(https://www.w3.org) CSS: centering things - World Wide Web Consortium (W3C)
People also ask
(https://developer.mozilla.org) Center an element - CSS: Cascading Style Sheets - MDN Web …
(https://stackoverflow.com) How to horizontally center an element - Stack Overflow
(https://blog.hubspot.com) 11 Ways to Center Div or Text in Div in CSS - HubSpot Blog
(https://css-tricks.com) Centering in CSS: A Complete Guide | CSS-Tricks
(https://dev.to) 3 Ways to Center an Element in CSS - DEV Community
(https://blog.logrocket.com) 13 ways to vertically center HTML elements with CSS

It gets even worse on harder queries. Bing/DDG is slightly better (MDN is first result and W3 second, but instant answer is unfortunately w3schools), but their image search is still crap. Yandex has great image search for some reason, but their (English?) text search is unimpressive. To see the real horror, try some really high CPC keywords on your phone, and see how the entire first page is just ads.

I presume the reason for this is because they realized they would make more money absolutely whoring out their search engine than trying to give people good results. At a certain point, you reach the end of the growth stage, and all that remains is extracting as much profit as you can from the decaying carcass of your once-great enterprise.

But the root cause of it all is these heavily SEO’d sites, right? It goes something like:

Make a shitty site with information (tutorialspoint, cyberciti, etc)
SEO it to hell
Site places above tldr pages, Arch Wiki, etc
Earn ad revenue

So if the root cause is the profit motive, why can’t you just filter out the sites with big profit?

If you load a site in headless Firefox, you can obviously measure how much ads it has. For example, load it once with and once without uBlock, and then compare e.g. loading times. There are also other things:

GDPR popups
excessive JavaScript
presence of modal dialogs requesting email

Then, you could get a score of how plagued the site it is. Combine this with the ordinary search rankings from e.g. DDG, Bing, YaCy, or whatever, and Bob’s your uncle.

Heck, you don’t even have to make a new search engine. Just make a block list for uBlock that hides links to plagued sites.

I don’t see how this could fall prey to Goodhart’s law. It’s pretty hard to pretend like your site doesn’t run ads, right?

A modest proposal to fix web search

yanmaani.github.io | 19 days ago |

7 comments

Michael Seibel on Twitter: "A recent small medical issue has highlighted how much someone needs to disrupt Google Search. Google is no longer producing hig

twitter.com | 19 days ago |

939 comments

How Not To Sort By Average Rating | Hacker News

While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer.

Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior. This gives you a posterior distribution over new ratings for an item. The benefit of a posterior here is that it lets you rank items by thinking in terms of the probability that the viewer would rank one item higher than another at random. By adjusting the magnitude of the alpha parameter to the dirichlet prior, you adjust your sensitivity to small numbers of observations. A small initial alpha will lead to rapid changes in the posterior upon observing ratings, whereas a large alpha requires a significant body of evidence.

The best part of the multinomial model with conjugate dirichlet prior is that the math is REALLY simple. The normalizing constant for the dirichlet distribution looks scary when stated in terms of the gamma function, but given this is the discrete case, just pretend everywhere you see the gamma(x), it is replaced with (x - 1)! and you will be ok.

Let me know if you would like to learn more, I would be happy to help.

You could also go one step further on the Bayesian path and infer even alpha from the data on your site, and introduce a loss function on your ordering.

Or you could do a semi-frequentist thing and simplify your math by using MAP estimates to rank. Basically instead of score = #pos/(#pos + #neg), it becomes score = (#pos+x)/(#pos+x + #neg+y), where you choose x and y to suit your needs. You could choose x/y in proportion to the average number of up/down votes on your site or you could even choose x/y in proportion to the average number of up/down votes of the author of the post. That would rank posts of trolls lower than posts of good users. By varying x and y you can tweak the strength of this effect. You can interpret this as giving each item by default x upvotes and y downvotes.

This certainly works much better than the formula in the article. For example if a post has 1 upvote and 2 downvotes, his formula will say that should be ranked lower than a post with 1000 upvotes and 2000 downvotes (because he's using the lower bound of the confidence interval). Obviously that's bad because while the first post could be a good one, we know for certain that the second one isn't. In general his method will rank posts with a low number of votes very low, even compared to posts with a high number of downvotes.

Absolutely! I just wrote a reply where I alluded to that, unfortunately I didn't refresh and see this post or I would have just plugged you instead.

The benefit of the Bayesian treatment here that I want to drill down on is how natural it is to adjust the prior to capture your beliefs about how items should be perceived in the presence of incomplete information. The frequentist approach is fine, but it does not provide such a pleasant, intuitive knob to tune.

Well, it depends on how you look at it. The Bayesian MAP approach is basically the same as the frequentist approach with made up data. Instead of using the maximum likelihood estimator p = pos/(pos+neg) you pretend that each new post already has some up/down votes by default, and then you use simply p = pos/(pos+neg). Seems like an even more intuitive explanation of the Bayesian knob to me! Rather than an abstract "alpha" to a "Dirichlet prior", you get something concrete (the number of made up votes). And you get a simple formula, which some would find desirable.

But I agree that the Bayesian approach is conceptually much cleaner. IMO the frequentist approach is just computational corner cutting for when the math in the Bayesian approach gets too involved, which is sometimes useful. What's nice about the Bayesian approach is that you state your assumptions and then it's just turning the math machinery. In contrast, in the frequentist approach the assumptions are interwoven and hidden in arbitrary choices in how the math is done (And then they claim that Bayesians are subjective! It's just that Bayesians admit that they are subjective. Frequentists try to hide the fact that they are more subjective in the math). The not so nice thing is that turning the math machinery is not always so easy and does not always produce fast algorithms. That's where maximum likelihood and friends come in, but I'd view them as an approximation to Bayesian methods.

Yes, the explanation of "imaginary votes" is by far the simplest way of thinking about the situation. Having mathematical formalism and a rigorously studied methods makes me feel OK about doing it though, otherwise I would be very uncomfortable with the approach :)

I love Bayesian statistics from a conceptual point of view, but the ease with which one ventures into the land of analytic intractability kind of puts me off more complex models. MCMC is such a clumsy tool (in addition to taking forever); variational methods look interesting to me but I don't really feel they are quite there yet.

I read the article, went "The ^&#$?" and came here to post this, more or less. It's amazing the $&#! people will get up to when they don't know Bayesian statistics.

That really isn't fair to what this is doing. If a bayesian wanted to lower bound the distribution, they would be using exactly the same formula, but with an explicit prior. The typical bayesian solution to this of the dirichlet prior has a significantly different convergence rate, 1/n vs 1/sqrt(n) so the resulting ordering is substantially different as well, and I would argue worse for this application. The method of setting a pessimistic prior cares much less about the number of votes for a popular item than does the method of lower bounding the distribution.

If you think most comments are bad, and you have an explicit prior that reflects this, then your ranking will update to "Hey, the comment was actually good!" at the optimal speed - no more, no less - given the rate of incoming ratings. Why would any other rate of updating be better?

Because there's a high variance on your estimate, so in a lot of circumstances just by variance you'll show users bad comments on top most of the time, rather than showing them the good comment that you are certain of.

In this context, the pessimistic prior is more of a hack than the confidence interval, because your mental model isn't that most comments suck. Your mental model is that most of them are ok. The pessimistic prior is just there to say, if 1000 people confirm that a comment is ok, we should show that over a comment for which we don't know whether it's ok or not yet, and this reasoning is much better modeled by a lower bound.

Here's a paper proposing a solution in that space, and which also compares itself to the article linked here (kind of nice to see... papers sometimes fail to cite stuff that's "only" posted online rather than properly published, even if the authors know about it and it's quite relevant): http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...

I emailed Miller a while ago to see what he thought of this reply, and he thought it also seemed like a reasonable approach. But, in his view, the criticisms of his method within their framework include things that in practice he sees as features. In particular, they view the bias caused by using the lower bound as a bug, but he prefers rankings to be be "risk-averse" in recommending, avoiding false positives more than false negatives. Of course, that biased preference could also be encoded explicitly in a more complex Bayesian setup, which would also be a bit more principled, since you could directly choose the degree of bias, instead of indirectly choosing it via your choice of confidence level on the Wilson score interval.

I don't think you have to resort to any overly complex machinery to achieve similar behavior. The simplest approach is to just use a non uniform prior. His pessimistic bound could be emulated by having an initial alpha that places more weight on low star ratings. The intuitive interpretation of that being "things are probably bad unless proven good" roughly. Another option would be to generate the prior based on the posterior distributions of other items. Just take the distribution of ratings observations for all products of a given type (perhaps only items produced by that company?) to get a sensible prior on a new item in that category.

The strength of priors here is that it is very easy to take intuitions and encode them statistically, in an understandable way. Taking the lower bound of a test statistic doesn't admit much in the way of intuition.

I agree the lower bound of a test statistic is a pretty indirect way of encoding intuitions. Somehow I tend to find loss functions the conceptually clearest way of encoding preferences about inference outcomes, though. But, in this case my feeling in that direction isn't very strong, and the priors-based solution seems fine.

There's a distinct difference in the asymptotic behavior though between the lower bound and the prior. The lower bound goes to the mean as 1/sqrt(n), the prior goes to the mean as 1/n.

That makes for a pretty significant difference in practice, and I'm not sure which is preferable.

You are absolutely correct that they are not mathematically identical. I struggled to word it in a way that would not mislead people, the distinction is important to emphasize.

It has a really big effect I think on the tone of what gets selected at the top. The lower bound prefers things that are preferred by a majority and very popular. The prior method prefers things that are completely un-objectionable and liked by just enough people to be sure of that. My hunch is that with the lower bound you get more interesting things bubbling to the top because it puts a stronger emphasis on popularity.

In all of these models, the giant variable that is completely ignored is the actual choice to rate something at all, versus skipping over it and reading the next one. That's a very significant decision that the user makes. The behavior of each of these systems w.r.t that effect will be the dominant thing differentiating them.

Can anybody replicate their results at Proposition 5? My tests contradict their conclusion, namely that the total score is not monotonic (i.e. when I test I do get that the total score is monotonic).

I'm too much of a Math idiot to understand that paper. Have you seen a code implementation of it anywhere?

> this is one of those cases where a Bayesian treatment is conceptually much clearer

Is there any other sort of case?

I am a beginning stats student. I took the AP class in high school, and have been doing independent work since. Predicting voters, playing with data sets, using a bit of Python to try to make general solutions, and other non serious but non trivial actions. Bayesian solutions have always required more work from me, even when they seemed the optimal way to solve a problem. I still don't entirely understand how to work with continuous distributions of hypotheses, partially because I lack the deep mathematical intuition at the moment. I don't blame the math, but I do think that much of the time the Bayesian treatment can be more challenging. I suspect that this is why non-Bayesian techniques remain so popular.

What he said! I'd like to add that a key to squeezing more out of NathanRice's post is the phrase "conjugate prior." Another totally natural thing would be to use a Gaussian prior & likelihood, then update the posterior as ratings arrive. This would take advantage of the ordinality of ratings as NR suggests. Bishop's Machine Learning book goes into this sort of stuff in more depth.

That's heavy on mathematical lingo and I confess I didn't understand it. Can you recommend an online resource that would explain what you just wrote in a newbie-friendly manner?

I would absolutely love to learn more. I've been trying to solve some novel NLP and machine learning problems lately but my lack of statistical knowledge is becoming apparent the further along I get.

Do you have any recommendations for a good introductory treatment of Bayesian statistics?

I want to second MacKay's book. I had a terrible statistics class in college. We spent the entire time looking up tables of p-values and t tests, without a very convincing explanation as to why. The entire topic was damaged for me from then on until I read MacKay online, and then bought the paper version.

His book starts from first principles -- simple ideas about probabilities -- and it builds a foundation for understanding Bayesian methods. And he explains, basically, why the p-values and t-test stuff is a bunch of crap, which was an immense relief.

So I don't need to flinch over the mention of "p hat" again?

Thanks for the resource, this looks fantastic.

I love it and I hate it.

Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works.

Why I hate it: What human can understand it?

I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to apply complex logic to everyday problems." Things like safety stock, economic order quantities, reorder points, make/buy decisions, etc.

But the designers of these systems overlooked one critical issue: these systems included humans. And as soon as humans saw that decision making formulas were too complex to understand, they relieved themselves of responsibility for those decisions. "Why didn't we place an order?" "Because the computer decided not to and I have no idea why."

I suppose the optimal solution is somewhere in between: a formula sophisticated enough to solve 95% of the problem but simple enough for any human's reptile brain to "get it". This isn't it.

>What human can understand it?

Lets start with Wilson's midpoint, since that's just high school math.

     def mid(upvotes:Int, downvotes:Int) = {
      val total = upvotes+downvotes+0.0
    
      val up = upvotes/total
      val half = 0.5
    
      val a = total/(4+total)
      val b = 4/(4+total)
    
      a * up + b * half
     }

So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes.

It should be very clear that if the total becomes large, a goes to 1 and b to zero. At that point you end up using the proportion of upvotes, just like Amazon.

Now, lets bring in the entire confidence interval.

     def wilson(upvotes:Int, downvotes:Int) = {
      val z = 1.96
      val n = upvotes+downvotes+0.0d
      val phat = upvotes/n

      val lower = (phat + z*z/(2*n) - z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
      val upper = (phat + z*z/(2*n) + z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
      (lower,upper)
      }

Using the author's data points, the results look like this:

     val itemVotes = List((600,400),(5500,4500),(2,0), (100,1))
     itemVotes.foreach( x => {
      val s = mid(x._1,x._2)
      val w = wilson(x._1,x._2)
      printf("Up:%5d\tDown:%5d\tMid:%.3f\tW:[%.3f,%.3f]\n", 
      x._1, x._2,s,w._1,w._2)
      })

       scala> Up:  600	Down:  400	Mid:0.600	W:[0.569,0.630]
        Up: 5500	Down: 4500	Mid:0.550	W:[0.540,0.560]
        Up:    2	Down:    0	Mid:0.667	W:[0.342,1.000]
        Up:  100	Down:    1	Mid:0.971	W:[0.946,0.998]

Basically, we prefer the lower bound of the confidence interval instead of the midpoint of that same interval.

The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it.

Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably correct. However, they're still going to field questions from merchants saying "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?"

Similarly, you may work in the acquisitions department of BigCorp and you need to purchase some equipment. When you prepare your report for Big Boss of your recommendations, how do you explain that you are recommending something which has a 4.5-star average rating instead of the 5-star average rated one?

The ratings may be 100% mathematically sound, but the users of the rating system are not the same as the producers of the rating system. So it's largely irrelevant if the producers made it perfect if the users don't understand what "perfect" means.

They'll only have to field those questions if they display the plain averages despite not sorting on them. If we've established that average rating is the wrong metric, why would you even show it to the user/merchant/whoever? Display the same metric that you sort by, and then the merchants will be perfectly happy with higher-rated items showing up higher on the list, and Big Boss will be happy that you've chosen the #1-recommended option.

I really don't think that solves the problem; you look at product X, it has 0 reviews, you give it a 5 star, now it shows 1 rating and 3.7 stars?

So you let people rate it with 5 stars, but then show it as 3.7 magic points. Magic points can be your proprietary ranking system.

You don't need to let them rate with stars and also show it as stars.

If I saw that happen, I would probably assume that the item was so popular that a bunch of other people had rated it concurrently. Unless the number of ratings were displayed simultaneously. There's a bit of a comprehension barrier in letting people know that the ordering score is not the same as the thing they're inputting, and what the relationship between them is. I don't know how to solve that problem (if I had to, I'd probably just not show the ranking numbers at all- the user already sees the sorting order, what do they need the raw numbers for?), but that's OK, 'cause I prefer to sidestep it- just don't use a graded rating scale like "5 stars". At least not unless you have a really good justification and explanation of what precisely the scale means. 5-star rating systems tend to result in large numbers of 5s and large numbers of 1s anyway, so you might as well just go with a binary like/dislike system. The fact that a ranking is a real number and your input is an "up"/"down" enumeration makes it clear that the ranking you see is not the same thing as the data that you put in. It could also be useful from a defusing confusion standpoint to make the ranking more complicated by mixing in an age factor (like HN does) or some other extra band of information. Then if a user doesn't immediately understand the relation between their rating and the final ranking, they can easily write it off as being due to some other factor that's not in their control anyway.

You can do all stars but if you say you need at least X people to rate before showing any. That's what many sites do.

I disagree that people won't be able to understand it - I propose they can't be bothered to understand it. Regular people do not give a shit about algorithms - either the computer/software works or it doesn't. Tech geeks regularly forget this simple fact.

Most people have other things they care about and all they want is to press a button and have an answer come out.

If its instantly clear how the answer is generated, they think about it for a very short while, agree and move on with their lives.

If it's not instantly clear how the answer is generated, they assume the computer is right and move on with their lives.

> "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?"

'Because you have fewer votes than your competitor'

I don't think it's that hard to understand.

Right. If you look at most-helpful ratios for product reviews on Amazon, you will see this. 500/600 helpful floats far above 10/10 helpful.

the article also misses the human factor. And intent.

for amazon, is it better to show the best product first or to help sell a few of the lacking-rating ones to generate more ratings and work out the uncertainty with real world data instead of crazy math?

I don't think it is a good idea to misuse your customers for that kind of testing.

I've dealt with people who have wanted a retraction of negative comments to their business on the web, I explained it didn't work that way as best I could, but there isn't much you can do. Same with this, if they are irate about a low rating, you play nice, but you don't expect them to learn the ins and outs.

Yeah, most merchants only really want to participate in rating schemes when it puts them above their competitors.

It's neither precise, nor elegant, and certainly not rigorous. On this approach, an item with 1 upvote and 2 downvotes will be ranked below an item with 1000 upvotes and 2000 downvotes, as one commenter pointed out; and in general, bright new items will almost never be presented to anyone. Search on "Bayesian" in the comments for the precise, elegant, rigorous solution below. I won't claim that it's easy to understand, but it's a lot easier to understand than the frequentist ad-hoc version.

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.

Amusingly the proposed solution in this blog post is almost line-for-line exactly the method Reddit uses to rank comments - proof, I think, that the method isn't "too complicated."

When you're working with a system managing orders, where there's a direct human interaction with the algorithm (like the top-level poster is), simplicity is somewhat important. When you're working with a system that's designed to look like magic to the end user, making the system more magical is often better, because it makes it less intuitive to game.

The math doesn't make the comments good. The math makes good comments easier to find, and buries comments with inadequate favor or confidence.

Besides, the math made no effort to write the sentences. ;)

I actually find issues with the "best" comment sorting, in some cases. I think it generally is the "best" sort, but in very popular posts it tends to break down.I often find comments with 5x or more upvotes buried down below the top few comments.

I think as comments get higher to the top, people start voting them down more, which might not be the cases for comments which are rising (theory: people reading further down in the comment section are possibly more thoughtful and more likely to upvote, while at the top the ADHD crowd might be more likely to knee jerk downvote?).

In any case, i've recently changed my default sort to "top" and feel it's an improvement.

I think you're overlooking a critical component of the Google equation: the supply.

Webmasters * constantly* complain about the impenetrability of the 'algorithm' and how it constantly changes for secret reasons.

So you're right in a way---the demand side doesn't care how you manage to curate supply, but supply is intimately aware of your curation and worries about being unfairly demerited.

what?

There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem?

Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not?

Hint: insurance/loan underwriters regularly end up in situations where the human component of a transaction may look different than the data, and react accordingly...

There's nothing wrong with the math.

I never said there was. In fact, I praised it as an elegant solution.

It just requires better educators to explain it, with analogies and metaphor.

You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000.

Why would you compromise the data/outcome in a hope to simplify the problem?

Actually, simplying the problem actually reduces the compromise when humans are involved. So, no.

...wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not?

Only if it made economic sense to do so. I am not going to replace 800 workers earning $12/hour with "the best people who understand the domain" because the computer suddenly has formulas that people don't understand. Experience has shown repeatedly that workers, at any level, simply stop caring when they feel powerless by "solutions" like OP's Wilson formula. That abducation of responsibility almost always far outweighs any incremental benefit that a more sophisticated but incomprehensible formula introduces.

Good points, nice discussion, but please, for the sake of this community, don't reply with words like "what?", "Further:", or "Hint:". You made your point without the snarkiness.

I'd imagine the right way to handle this would be Reddit style: Implement the naive solutions (i.e. sorted by average score) alongside the top-performing solutions.

Add in a UX that makes it obvious that there's a choice to be made and allows the user to easily evaluate the results of the naive solution alongside the optimized solution, and end users who are themselves evaluated on overall performance metrics will tend to choose the more effective solution whether they understand it or not.

The addition of metrics to allow for side by side evaluation of the simple strategy versus the optimized strategy is key - someone choosing the unintuitive yet superior strategy will likely have to explain themselves to management at one time or another.

Aan interesting parallel is NFL onfield playcalling decisions. There are plenty of instances where coaches make choices that are statistically unsound but they reinforce the image of a tough, responsible coach. Conversely there are coaches who play by the odds and get lambasted when their decision doesn't pan out.

Workers who feel powerless by sophisticated solutions need to learn the solutions, and not be intimidated by them.

Yes, assembly line workers need simple steps to do a job. But that of course isn't who this is referring to. In your example managing inventory isn't an assembly line, checklist job.

You're not replacing 800 workers with the best. In your example, you wouldn't have 800 people managing inventory. You replace your inventory manager with someone who understands the math (if you want to manage your inventory that way)

Sufficiently advanced sophistication is indistinguishable from obfuscation. And obfuscation is a problem that's not ideally solved by hiring smarter people.

I think that's the point edw519 was trying to make.

> It just requires better educators to explain it, with analogies and metaphor.

So... anyone want to take a stab at explaining that equation to those of us who don't really get it?

If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom.

[1] http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...

Wilson's 1927 paper is freely available [1]. I can't say I have brushed up on my statistics enough this decade to verify the math but the basics as I read them are as follows:

- You have a normal distribution (bell curve) of data points, in this case quality scores.

- You wish to sort these points based on their respective vote totals. Any given data point has pos positive votes out of n total votes for that item.

- You have a confidence interval, e.g. 95%. This confidence is expressed in terms of the bell curve, so a 95% confidence is within 1.96 standard deviations of the mean [2].

- You have a Ruby function accepting the aforementioned n, pos, and confidence variables and returning a decimal value representing the normalized confidence_interval_lower_bound, that is the quality score that our input data point has a 95% chance of meeting or exceeding.

- Given a set of data points, evaluate the ci_lower_bound for each, and then sort them accordingly. The results will give you a best-guess sorting that accounts for the fact that some data points will have more votes cast for/against them than others.

[1] http://www.med.mcgill.ca/epidemiology/hanley/tmp/Proportion/...

[2] http://en.wikipedia.org/wiki/1.96

With zero knowledge of statistics, here's how to think about it - Wilson wants to construct some score. That score can be as low as 0 and as high as 1. So he wants some interval [x,y]. The center of that interval would obviously be c = (x+y)/2. Wilson first decides what c to pick. So Wilson says, the center c must be decided by the proportion of upvotes. But then he thinks, c must also be close to a half. So he says, okay, lets figure out c using some sort of an average. So he chooses two weights a and b. The weighted average would then be a times half plus b times the upvote proportion. He chooses those weights in such a fashion that if you have lots of data, one of the weights vanish & the other becomes unity. So the weights matter only if you have too few upvotes & downvotes.

Having figured out the midpoint c, Wilson has to actually figure out the lower bound x and upper bound y. Now he draws a distribution centered at c....ok so at this point you would need to know what is a distribution, and why you would need one, whether that distribution has a skew & whether its homoskedastic & so on...which is stats 101, so I won't go there. But if you've gotten this far, you should be able to atleast see the intuition behind Wilson's procedure.

Say the 'acutal' rating for an item is p. (ie, if you got an infinite number of people voting, the ratio of upvotes to total votes is p).

Now say your users vote, and they upvote with that probability p. The number of upvotes k you get out of n votes will follow a binomial distribution B(k;n,p). The binomial distribution has mean np and stdev sqrt(n p(1-p)), and is very close to gaussian in shape. Since the stdev is a rough measure of the 'width' of the distribution, common way to describe the error is (mean +/- lambda*stdev), where you can tune lambda to your desire. If you increase lambda you get a wider confidence interval, and therefore more certainty that a measurement will be within that confidence interval.

Now, say you measure k upvotes out of n votes. You can divide by n to get p0, your estimated rating based on those votes.

An easy estimate for the error of this measurement is to assume that p0 is approximately correct and equal to p. Then the expected number of upvotes would be n p0 with stdev(n p0(1-p0)). Divide this by n to get the fraction of upvotes, to give a final estimate for p of p0 +- lambda sqrt(p0(1-p0)/n)

Now, your estimate (p0) of p is not quite right, and therefore your estimate of the error (which depends on p0) is not quite right either. The wilson score attempts to correct for that. We don't know p, but imagine if we did, we would expect any measurement p0 to be in the range p +- lambda sqrt(p(1-p)/n). That is, we expect

        abs(p - p0) < lambda sqrt(p (1-p)/n)

If you solve this equation for p in terms of p0, you get a formula given in the article, ie, the confidence limits for p given p0.

I thought he was talking about the end user. They may not understand why something with 209 thumbs up and 100 thumbs down ranks better than one with 5 thumbs up (100% positive).

I hate it exclusively. It's clearly better than (1) and (2), but...

I talk about this with some of my med school friends that are interested in/want to create/hate/fear automated diagnosis. At one level, having appropriate statistics to make use of a wealth of prior experience, worldwide prevalence, epidemiological data, &c is basically a requirement for the future of proper healthcare.

It's also an obvious terrible mistake to ever convince people that these statistics are anything besides a decision making tool.

I think the right usage of statistics is to enlighten and confuse simultaneously, not to "answer". They should provide analytical depth to a decision, never an escape route.

For this reason, I think proper statistical application is an intersection between mathematics, computer science, and UI design.

> I think the right usage of statistics is to enlighten and confuse simultaneously, not to "answer". They should provide analytical depth to a decision, never an escape route.

You're right, but I would go further. I would argue that when statistics are presented as "the answer," in many cases they are being abused. A good scientist recognizes the limits of his dataset. Like how a computer program can only do what it is written to do, statistics can only tell you what is in the data, they can't tell you what is _not_.

No matter how much statistical data you reveal, you still need to present the results in some particular order. And this formula is the best ordering function I know of for things that are rated.

I think the situation is much more informatively modeled by a partial order though. You can induce a total order on a partial order and you can use this formula to do so conservatively, but maybe the real way to solve the problem is to create an intuitive way for users to appreciate partial orders.

Operations research and hard systems modeling are neat. The only gripe I have is that often times the models make simplifying assumptions which hold tenuously at best. One example of this is the Gaussian copula used in VAR. In actuality, neither the marginals nor the copula are Gaussian in the large majority of cases; they ARE stable distributions, but the tails can be absolutely HUGE. The Gaussian distribution is attractive because it is well behaved, and in most cases we don't have enough observations to properly understand the full tail structure of the underlying distribution. As a result, you get quants that make a model and call a small subset of points (which are completely valid) "outliers". Then when we observe the tails of the distribution, things fall apart because over-leveraged bankers made decisions based on the notion that the model accurately reflected reality.

You have overlooked a critical point: no one has to mess with this (otherwise quite simple) formula anymore, except the programmer, and even he already got a working piece of Ruby code.

»The computer decided not to and I have no idea why

This just means you need a better notification system, not a dumber model. I am fairly sure Netflix and Pandora use complicated recommendation algorithms - we don't complain because the complexity is abstracted away behind labels like "dark cerebral thrillers" or "songs with rhythmic gyrations like Pretty Lights".

Right, the solution here is that if you're going to sort by Bayesian average or Wilson score confidence interval or something besides what we're familiar with, reframe it to avoid my "something's wrong" detector. Hence the fact that people accept "trending" or "popular" but if I sort by "rating" and the results don't make sense because I expected an average, I don't think "I should probably understand this algorithm". I think "this site is broken and dumb".

And as soon as humans saw that decision making formulas were too complex to understand, they relieved themselves of responsibility for those decisions. "Why didn't we place an order?" "Because the computer decided not to and I have no idea why."

And somehow the solution isn't to have the computer output a detailed description explaining why it arrived at the decision that it did?

Math is hard, let's go shopping!

If you think this is too difficult then maybe you shouldn't try to design scoring/voting systems.

> "Why didn't we place an order?" "Because the computer decided not to and I have no idea why."

This may or may not be a problem. Remember, human intuition evolved to be able to give a fast answer to any question in the face of insufficient data, not an optimal answer to specific questions with sufficient data. In many domains, once you have appropriate data and the means to statistically analyze it, an algorithm using such analysis can reliably outperform human experts - and in some cases, the problem was the political one of how to get the human experts to accept this, take their hands off the wheel and stop second-guessing the algorithm.

In other words, if you are dealing with such a domain, you may be better off to count your blessings, let the computer do its job and save the humans for tasks that can't be handled by a formula.

To get a general idea, you can compare this formula with the simple version, that is * mu - 3 sigma* (be careful when the number of samples is small), where mu is the average and sigma is the standard variation, and 3 is a parameter that depends on the confidence (~93%). When the number of samples (n) is big, then the two formulas give similar results.

The way I think about it is this: comprehensibility is a feature. You have to weigh it against accuracy, effectiveness, etc., but it should be taken into account.

Meh. "Why did you return the list of reviews in this order?" - "Here's the link to Wikipedia explaining it".

The formulas aren't "too complex to understand", the people are being too lazy to take the time and understand them. No reason to dumb it down and ruin it for the rest of us.

Original author here. For the academically inclined, there is a critique of this approach in this paper:

http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...

Of course, I think the authors miss the point of the algorithm, since I basically wanted a system that is one-sided (i.e. false negatives are OK but false positives are bad).

Also, if you deal with more than two outcomes you might be interested in multinomial confidence intervals, described here:

http://www.math.wsu.edu/faculty/genz/papers/mvnsing/node8.ht...

The application to 5-star systems is not straightforward, since it's not clear to me how stars relate to each other. Is it a linear scale? Are they discrete buckets? Or maybe we want to use Tukey's froots and flogs? I'm not sure.

By the way, I'm coming out with a stats app for Mac soon that implements this algorithm and much more. Drop me your email address if interested:

http://wizard.evanmiller.org/

I appreciate people who take the time to apply math to things in the real world, and share it with non academic crowds. Thanks for that.

5 star rating systems are obnoxious. From a mathematical perspective, if you treat them in an ordinal fashion they are poorly behaved, and if you treat them categorically, you lose the relationship between stars. There seems to be some popular movement towards binary rating systems, and I think that is great. Not only do people tend towards binary rating behavior in the real world (only rating a movie they thought was very good or very bad) but they admit a much cleaner mathematical treatment.

> 5 star rating systems are obnoxious. From a mathematical perspective, if you treat them in an ordinal fashion they are poorly behaved, and if you treat them categorically, you lose the relationship between stars.

Helping out a friend with a statistics test, I recently read up about the Wilcoxon Signed Rank Test[1]. Now this one is intended to get a p-value for experiments with "before" and "after" measurements, but what I got the idea it's trying to do, is to use the ranks of a not-very-normal behaving random variable, turn that into a summation of lots of things, so due to the central limit theorem you can treat it as a normal distribution again.

Though thinking about it, in this case it's the rank we're after, so maybe it's not useful at all. But it gives an interesting idea about the tricks you can pull if your input data isn't quite the sort of type that you can analyse very well.

[1] http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test

I really appreciated the post. Unfortunately we're using 5 stars, and need to do it a bit less wrong.

The main thorniness of 5 stars is that you have to answer the question of what the difference in star ratings actually mean. Is going from one star to two stars the same as four to five? Probably not, based on the way users rate, which means an algorithm like the arithmetic mean that treats them the same is wrong.

Personally, I think it's very reasonable to treat rating stars the way we should treat grades: as ordinal data, where we know that a higher rating is better but assume nothing beyond that. The difference between an F and a D is not the same as an A and a B, and the same is likely true of 1-2 stars and 4-5 stars.

I have made an attempt at applying this idea to the Ubuntu App Store's rating algorithm. I'm very much interested in comments. https://bugs.launchpad.net/ubuntu/+source/software-center/+b...

"we know that a higher rating is better."

Typically, you do not know that, especially if you are comparing across reviewers. Some will only score zero and 5 stars, other will have 10% two stars, 80% three stars, and 10% four stars, yet others will have 10% three stars, 80% four stars, and 10% five stars.

If you have sufficient data (rare), it may be possible to (somewhat) correct for that. IIRC, this was something that helped in winning the Netflix challenge.

For example, http://www.netflixprize.com/assets/GrandPrize2009_BPC_Pragma... models the assignment of stars as two parts:

- modeling the user's appreciation of the movie - modeling how the user translates his appreciation to a star rating

"where we know that a higher rating is better but assume nothing beyond that"

Problem with that is that you throw away information with that assumption. You do know that 2 star scores are very unlikely to be about very good items.

Mathematician not a statistician...

Would it be reasonable for 5 stars to normalise the data? Should star ratings be on some distribution, for instance?

In the binary case the usual treatment in statistics is to use the logistic function (and logit) to work with real numbers, then transform back into probability space as the last step.

This is a little flakey for ordinal numbers, and the usual treatment is to use a learning algorithm to find a mapping from real numbers to ordinal values, either explicitly (if you need a "score") or implicitly. Support vector machines, radial basis functions and neural networks are typically used.

There are a lot of comments complaining about how complicated the math is. This shouldn't be all that hard to understand.

The assumption is that there's some constant p underlying probability that a random person will rate a given thing positively. If we observe, for instance, 4 positive and 5 negative reviews or votes, there's a probability distribution (known as a Beta distribution) which tells us what the possible values of p are given the votes we observe: p^4 (1-p)^5. graph: https://www.google.com/search?q=x%5E4+(1-x)%5E5%20from%200%2...

Now if we observe 40 and 50, respectively, the curve looks like this: https://www.google.com/search?q=exp(20+%2B+40+log(x)+%2B+50+...

(I had to do it in the log domain because Google's grapher underflows otherwise -- the 20 is just to make the numbers big enough to graph. The more correct thing involves gamma functions and that just gets in the way right now)

The more you observe, the more sharply peaked the likelihood function is. The funky equation in the article is an approximation to the confidence interval of that graph -- 95% of the probability mass is said to be within those bounds.

It's not a great approximation, for one because the graph is skewed (try it with 10/50) and it assumes that the mean is exactly in the middle of the confidence interval. The correct computation involves the inversion of a messy integral called the incomplete beta function. Scipy has a package which includes betaincinv which solves this more exactly:

>>> import scipy.special

>>> scipy.special.betaincinv(5,6, [0.025, 0.975])

array([ 0.18708603, 0.73762192])

would be the 95% confidence interval for 4 positive and 5 negative votes;

>>> scipy.special.betaincinv(41,51, [0.025, 0.975])

array([ 0.34599562, 0.54754792])

for 40 and 50, respectively.

[edit: apologies, I had to run and get ready for work -- I didn't really have time to make this very comprehensible; but i just now fixed a bug in my confidence interval stuff above]

Or, in layman's terms, "If we rounded up the entire population and forced every single person to carefully review this item and issue a rating, what's our best guess as to the percentage of people who would rate it positively?"

And to make the description slightly more accurate, at the expense of more complexity: "What number are we 80% certain the approving percentage will exceed?"

Very interesting. So would you say developers should probably use the incomplete beta function, rather than Ev's method? Or is it too computationally expensive?

I haven't investigated it in depth -- quadrature over a single variable as this is can be pretty quick to compute. Not sure how scipy does it.

Anyway, I personally think 95% confidence intervals are a crutch. The correct Bayesian approach is to consider two items, each with their own up and down votes, and integrate over all possible values for p1 and p2 (being the underlying probabilities of upvotes for item 1 and 2, respectively) over the observed data, and compute the likelihood of superiority of p1 over p2.

How to turn that into an actual ranking function? No idea. I doubt it would work, but you could compute against a benchmark distribution (i.e. the uniform 0-1 distribution).

If you do that, it probably turns out that your ranking function is the mean of the Beta distribution, which is simple: (U+1)/(U+D+2) where U and D are the upvote/downvote counts [note: we started with the prior assumption that p could be anywhere between 0 and 1, uniformly]. Basically, the counts shrink towards 1/2 by 1. This is a hell of a lot less complicated, and it achieves the goal of ranking different items by votes pretty well with more votes being better.

While it is good to look at these sorts of mathematically rigorous algorithms, I think I would be frustrated if it was used everywhere. Or, well, maybe not me perhaps, but a non technical user.

The beauty of the second algorithm for rating products is that it is straightforward. Having never seen it before I can deduce that 5 stars come before 4 stars and more reviews come before fewer. If I want to skip ahead to the 4 stars I know what to do. I can internalize the sorting algorithm easily. And as a user, understanding the order items are presented to me is important.

If Amazon were to use the last algorithm and present items in that order (assuming we accounted for the 5 star vs positive/negative issue), it would like a random order to most users and would be frustrating.

So I guess what I am saying is that this algorithm is very clever, but in some cases, it may be too clever. Sometimes you just want to keep it Simple Stupid.

The second algorithm, in my experience, is too simple, though. When browsing Amazon I'm pretty regularly annoyed by an item with one 5* review appearing ahead of an item with hundreds of 4* and 5* reviews.

One simple fix would be to avoid calculating an average until a minimum number of ratings have been given. But I do think the statistical way is lovely. If I were Amazon I'd give it some kind of snappy trademarked name and push it as a feature.

Instead of displaying stars, Amazon could display a percentage, which under the hood represents the Wilson confidence number. It would be totally intuitive to browse: first come all the 100% items, then the 99's, and so on.

You can't use Wilson's confidence with a star-rating system. Wilson's only works for binary systems.

Instead you could use a weighted baysian rating:

br = ( (avg_num_votes * avg_rating) + (this_num_votes * this_rating) ) / (avg_num_votes + this_num_votes)

How is it a problem if the five-star reviews display first, then the four-star, and so on?

The point is we have to determine how to define a five-star item, a four-star one, etc. Currently, an Amazon item's star value is the average of the star values of every review. The author is saying that that's a bad way to compute the item's star value. The author would argue an item with only two reviews that are both fives should have a lower star value than an item with 400 fives and 1 four. We typically associate stars with the averaging algorithm (i.e. we define an item's star value as the average of the star values of its reviews), so it might help if we do away with the notion that each item has a star value, and just think of this as saying an item with 400 reviews of 5 stars and 1 review of 4 stars should be shown before an item that just has 2 reviews of 5 stars.

Currently, when we see an item's star value, we think of it as an indicator of the quality of the item. But if it's just the average of the star values of every review, the author would argue that we're not going to get an accurate indicator of quality. The author argues that whether the quality indicator of an item is expressed in stars or percentages, that value should be determined by the third algorithm, not the second, and that the order the items are shown in should be the result of sorting those quality indicators.

But how is Simple Stupid in the Amazon case a better output for the user? Do you, as an Amazon shopper, really believe that the item with one 5-star review is a better bet for you than the item with 580 reviews and an average of 4.5-stars?

I don't, but I can intuitively grasp that a 5-star item with 2 reviews is not reliable. Since I understand how the sorting works, I know I have to jump to the 4.5 star items in order and check how many reviews that item has and if it also has a small number then I will jump to the 4 star items.

The point is, I understand the sorting order and can manipulate them if I am not satisfied with what is presented to me. Having a very esoteric algorithm is a risk. Maybe you'll present just what the user really wanted. But if you get it wrong they will be lost to do anything about it. I tend to dislike systems that leave users helpless when something goes wrong.

I implemented this in a rating system once. Got multiple bug reports, people complained that the system calculates averages wrong, because there are two ratings and the average is obviously not the number they are seeing.

Why show the second figure to the user? They don't need to see the calculation you made to determine the sort.

Just sort the objects in the order determined by this formula and only show the ratings given by users in the interface.

This was a system where I was supposed to display a star-rating (1-5 stars, with fractional stars as well) for each item.

The people reporting the problems were the users, and yes, this was bias, as they expected averages. That was exactly my point — while the statistics behind this method are sound, it is not what people expect. Building systems that don't do what people expect is difficult.

This is mildly amusing because above in this thread the exact opposite comment/response happened; someone said they used it for sorting but showed the mean next to items, and got bug reports that 4.5 was sorted above 5.0. The reply was that he should have just exposed the numbers that they are sorting by as the visible rating in the interface.

I'm curious, who filed the bugs - the users of the system or the fellow builders of the system? If you rolled back this algorithm based on the bug reports, it sounds like you might have ended up listening to the prior bias of the people on your team, rather than the feedback of the people meant to use the system.

It's well explained, informally. The giant equation sitting there without clearly defined parameters is mostly just showing off though. The final "QED" solution that you put at the end of a paper is not the proper form to introduce a concept.

But... so what? Amazon and Urban Dictionary are hardly failing in the market due to their "incorrect" score sorting. The whole problem is a heuristic, it's not amenable to rigorous treatment no matter how many giant equations you club your audience with.

It would be instantaneously annoying to someone, who would downvote it and push it off the front page. So the net effect is that the frequency with which you saw "low vote garbage" would increase, but probably not by much. And IMHO a reasonable argument can be made that this is a good thing, because it increases visibility for new posts.

For those who do not understand the Wilson algorithm, see this post which was on HN recently, explaining how it works in a little more detail: http://amix.dk/blog/post/19588

(I agree with other commenters that it is complicated and lacks common sense to average users, but I feel like I have a general understanding of the concept thanks to the above link)

Every time I see that page, I see the equation, I read statistical terms and I get overwhelmed. I use PHP so I have no pnormaldist. Would love to use it for some random page I run.

I was overwhelmed too (here's simple wrong solution #1, here's simple wrong solution #2, here's incredibly complicated but more correct statistical solution #3 that you may not understand) but since this is relevant to my day job and we use PHP, I did find this:

http://www.derivante.com/2009/09/01/php-content-rating-confi...

I'm not a statistician so I can't speak to the correctness of this implementation.

"if you don't have a statistics package handy or if performance is an issue you can always hard-code a value here for z. (Use 1.96 for a confidence level of 0.95.)"

Here's an even simpler way to think about it: it's the left point of the standard 95% confidence interval from the Central Limit Theorem plus a hack for small sample sizes. The Wikipedia page says the hack is almost equivalent to estimating p = (X+2)/(n+4) i.e. assuming each item starts with two upvotes and two downvotes.

I've put some thought into metrics as well. A few other alternatives suggest themselves:

- Considering the standard deviation of ratings. On a 5 point scale, an item that rates 3 because ratings are split between 1 and 5 votes, differs from one that gets mostly 3 votes. The latter is a middlin' fit for anyone, the former has an enthusiastic but niche audience. If you're looking at sales, the former can be a valuable product if properly marketed.

- An item that gathers few votes regardless of favorability ratings can exhibit multiple problems. One is that it isn't well marketed / publicised, or known. Another (particularly on content sites) is that there's very likely a sampling bias (mutual admiration society / negging attack / vote stuffing). I've tended to favor systems which take into account the total volume of voting, generally on a ln(n) basis, though not out of any particular statistical rigor. As an implementation, you'd start with a 5 point Likert score, then multiply by, say, ln(n+1) (avoiding a zero multiplier on a single vote).

- The pattern of ratings over time and space (IP or geographical) may reveal both opportunities for marketing and/or issues with your ratings system. Since any effective quality proxy will be abused, you've got to be sensitive to the latter.

The Wilson score is an improvement over multiple other methods. It still does assume a relatively unbiased estimator and rating behavior. My feeling and experience is that excess reliance on any one metric is likely to cause problems -- reality is multidimensional, metrics for assessing reality should be as well.

There's also the question of whether or not you want to make specific recommendations for an individual, or general recommendations for a population. In the former case, correlating other rankings or behavior may give a better fit (and the Wilson score may still be useful).

Though for a suitably specific goal (marketing, suitability, revenue potential) a single encompassing metric may work.

The one example that I've found of a good site that does really good average ratings is steepster, it picks teas that you have previously rated and indicates the rating you gave to them. This way the users rating is much better and will give you a much more meaningful mean.

Doesn't seem to have improved the relevance of the definitions:

"USA: The only country keeping penguins from coquering [sic] the Earth"

This is awesome. This is perfect for what we need for our startup. We are going to use this. We won't need to worry about the negative aspects listed in these comments due to our use case. Wow. Thanks, HN. :)

But shouldn't the solution (formula) be "simply elegant"? eBay seems to be on to something with: positive/positive + negative rating system. The user knows how many data point are in the pool which over comes the one positive rating gets five stars. Much in the same way http://demanddriventech.com/home/solutions/replenishment/ has come up with a "simply elegant" formula for supply chains that is human understandable and effectively solves the problem.

My favorite part was looking at the code and seeing a variable named `phat', and then looking up at the equation to find `p^' (p-hat).

What I don't get is how in 2012 sites like Amazon are still making this mistake. Amazon is a company that, much like Google, spends millions analyzing user behavior and trying to optimize the workflow (checkout, in their case).

This has been the number one complaint I have against Amazon for the past 10 years. And they haven't done a think about it?

It seems to me there's still a problem at the point of data collection. Not all +1's are equal.

This algorithm needs to be paired with another algorithm that weights each plus one according to each user's ability to plus one something that gets a lot of plus ones.

I couldn't hope to do the math for something like that, but I'd sure like to talk to someone that could.

When I have the whole body of reviews readily available, I like to just do a Bayesian average. Mix in the average number of reviews at the average review score to keep small data sets from skewing results.

Haha, I don't know why, but I laughed when I got to the 3rd formula. It was like the punchline to a joke.

I've always thought about this, and to me, a very simple (though slightly inaccurate) solution would be to sort using this formula:

(TotalScore - 1) / MaxPossibleScore

Such that (using the Amazon examples from the article):

((2 * 5) - 1) / 10 = 9/10 = 90%

((100 * 5) + (1 * 1) - 1) / 505 = 500/505 = 99%

we use this algorithm for our office ping pong game tracking system. it's great because the person who just plays one game and wins doesn't get bragging rights.

The last time was about a month ago IIRC. Can anyone explain why this keeps happening (and why people keep giving karma to those who just repost month old HN links)?

Probably because some haven't seen it yet and/or because people love discussing this topic.

That's why a friend of mine joined the army engineers.

As a civil engineer working for a local city he might be involved in a 10year process of approvals to add a freeway on ramp. Where most of his job would be checking that an army of subcontractors were all doing things to code - not that they were doing things well, just to the written requirements

In Afghanistan if they want a road or a barrier he basically finds somebody lower rank points at a bulldozer and tells them to do it.

An interesting point he made was building a simple village clinic with a clean water supply that would save lives for a few days work and a few $1000. At home he would be involved in a multi $100M, 20year project for a new hospital where most of the money would go into pretty decoration and parking structures and would probably end up costing lives compared to the existing old hospital that was working perfectly well.

Date: 9:12 AM Wednesday, April 4, 2012

From: the boss

To: dev3

Subject: URGENT - front page showcase selection broken!!

Body: Hey bro, I was looking into it, and our ratings average equation is totally busted and products with just a few ratings are hogging space from proven winners when it's just a sample bias. This is costing us money and needs to be fixed NOW.

I'd like this up before our morning meeting so I can boast about it and you'll get credit too, as this should massively increase our conversions right away by putting BETTER products right on the front page.

this should get you started: http://evanmiller.org/rating-equation.png

I'm sure you'll figure it out. If you could do an A/B test for bragging rights too that would MASSIVELY rock. Thanks!!!

Rock on,

Boss

That's the same thing as the average rating - works great if everything has about the same number of ratings.

The problem with star ratings is that they have nothing to do with measuring approval. They are a form of social inclusion mechanism to give the rubes the erroneous sense that someone cares about their opinions. It is done to attract users, not to guide them.

How Not To Sort By Average Rating

news.ycombinator.com | 26 days ago

AI Research Trends:Sort ArXiv papers by its popularity on social media sites

mavenlin.github.io | 29 days ago |

5 comments

DuckDuckGo is working on a privacy-focused desktop browser - The Verge

theverge.com | 32 days ago

Web Browser telemetry - sizeof(cat)

Web Browser telemetry

This is not an article about which web browser is the best (there is no such thing) but an article about which browser has the most connections after a fresh install. The results shouldn’t be a surprise, really.

Of course, not everything is telemetry per se, some of the connections are from the ‘New Tab’ pages that include Youtube videos but I strongly believe that is the same thing: the software leaks information about you and/or your computer without you actually accepting the data exchange (because you don’t have a way to accept or deny the connection since it’s the first time you’re running the browser).

All web browsers tested are the latest versions available today (December 19, 2021), test device is a MacBook M1 laptop (so, macOS but the results should be similar regardless of the operating system) with a clean profile, all network connections blocked, application-level network connection whitelisting and only Little Snitch installed. So it’s actually Little Snitch that does all the heavylifting, I’m just the robot that is parsing the data.

Running Microsoft Edge on macOS was extra challenging because the download comes as a .pkg installer with preinstall and postinstall scripts, so I just extracted the .app from the package. Beware the preinstall and postinstall scripts leak additional information .

Mozilla Firefox

Direct download URL
Version: 95.0.1
Connections: 15

firefox.settings.services.mozilla.com on TCP port 443
contile.services.mozilla.com on TCP port 443
push.services.mozilla.com on TCP port 443
shavar.services.mozilla.com on TCP port 443
detectportal.firefox.com on TCP port 80
location.services.mozilla.com on TCP port 443
www.mozilla.org on TCP port 443
accounts.firefox.com on TCP port 443
normandy.cdn.mozilla.net on TCP port 443
r3.o.lencr.org on TCP port 80
contile-images.services.mozilla.com on TCP port 443
ocsp.digicert.com on TCP port 80
tracking-protection.cdn.mozilla.net on TCP port 443
safebrowsing.googleapis.com on TCP port 443
ocsp.pki.goog on TCP port 80

Brave

Direct download URL
Version: 1.33.106
Connections: 7

laptop-updates.brave.com on TCP port 443
variations.brave.com on TCP port 443
go-updater.brave.com on TCP port 443
componentupdater.brave.com on TCP port 443
brave-core-ext.s3.brave.com on TCP port 443
crlsets.brave.com on TCP port 443
ftx.com on TCP port 443

Ungoogled Chromium

Direct download URL
Version: 92.0.4515.131
Connections: 0

Vivaldi

Direct download URL
Version: 5.0.2497.32
Connections: 13

239.255.255.250 on UDP port 1900
mimir.vivaldi.com on TCP port 443
update.vivaldi.com on TCP port 443
downloads.vivaldi.com on TCP port 443
vivaldi.com on TCP port 443
www.gstatic.com on TCP port 80
www.youtube-nocookie.com on TCP port 443
fonts.gstatic.com on TCP port 443
www.google.com on TCP port 443
yt3.ggpht.com on TCP port 443
i.ytimg.com on TCP port 443
update.googleapis.com on TCP port 443
edgedl.me.gvt1.com on TCP port 80

Tor Browser

Direct download URL
Version: 11.0.2
Connections: 0

Opera

Direct download URL
Version: 82.0.4227.33
Connections: 21

autoupdate.geo.opera.com on TCP port 443
sitecheck.opera.com on TCP port 443
redir.opera.com on TCP port 443
239.255.255.250 on UDP port 1900
speeddials.opera.com on TCP port 443
sd-images.operacdn.com on TCP port 443
weather.opera-api.com on TCP port 443
features.opera-api.com on TCP port 443
exchange.opera.com on TCP port 443
android.clients.google.com on TCP port 443
extension-updates.opera.com on TCP port 443
merchandise.opera-api.com on TCP port 443
desktop-dna.osp.opera.software on TCP port 443
download3.operacdn.com on TCP port 443
update.googleapis.com on TCP port 443
download5.operacdn.com on TCP port 443
mtalk.google.com on TCP port 5228
addons-extensions.operacdn.com on TCP port 443
edgedl.me.gvt1.com on TCP port 80
addons.opera.com on TCP port 443
get.geo.opera.com on TCP port 443

Google Chrome

Direct download URL
Version: 96.0.4664.110
Connections: 9

239.255.255.250 on UDP port 1900
accounts.google.com on TCP port 443
clients2.google.com on TCP port 443
tools.google.com on TCP port 443
clients2.googleusercontent.com on TCP port 443
edgedl.me.gvt1.com on TCP port 80
clientservices.googleapis.com on TCP port 443
www.googleapis.com on TCP port 443
www.gstatic.com on TCP port 443

Chromium

Direct download URL
Version: 99.0.4775.0
Connections: 12

www.google.com on TCP port 443
239.255.255.250 on UDP port 1900
www.gstatic.com on TCP port 443
accounts.google.com on TCP port 443
apis.google.com on TCP port 443
ogs.google.com on TCP port 443
optimizationguide-pa.googleapis.com on TCP port 443
ssl.gstatic.com on TCP port 443
fonts.gstatic.com on TCP port 443
play.google.com on TCP port 443
update.googleapis.com on TCP port 443
edgedl.me.gvt1.com on TCP port 80

Microsoft Edge

Direct download URL
Version: 96.0.1054.62
Connections: 21

nav.smartscreen.microsoft.com on TCP port 443
239.255.255.250 on UDP port 1900
config.edge.skype.com on TCP port 443
ntp.msn.com on TCP port 443
www.bing.com on TCP port 443
edge.microsoft.com on TCP port 443
arc.msn.com on TCP port 443
self.events.data.microsoft.com on TCP port 443
smartscreen-prod.microsoft.com on TCP port 443
assets.msn.com on TCP port 443
img-s-msn-com.akamaized.net on TCP port 443
api.msn.com on TCP port 443
bing.com on TCP port 443
browser.events.data.msn.com on TCP port 443
img-prod-cms-rt-microsoft-com.akamaized.net on TCP port 443
msedge.b.tlu.dl.delivery.mp.microsoft.com on TCP port 80
r.bing.com on TCP port 443
r.msftstatic.com on TCP port 443
deff.nelreports.net on TCP port 443
go.microsoft.com on TCP port 443
microsoftedgewelcome.microsoft.com on TCP port 443

Apple Safari

Direct download URL (good joke, heh?)
Version: 15.2
Connections: 6
Note: Safari does some tricks by delegating connections to additional daemons, like parsecd or com.apple.safari.safebrowsing.service.

token.safebrowsing.apple on TCP port 443
www.apple.com on TCP port 443
securemetrics.apple.com on TCP port 443
securemvt.apple.com on TCP port 443
gateway.icloud.com on TCP port 443
api-glb-euc1b.smoot.apple.com on TCP port 443

Conclusion

I won’t draw a conclusion because this is not the Washington Post. But I do believe that’s a huge amount of connections for some of the browsers since it’s actually the first start after installing. Huge.

Testing various web browsers to see what network connection they open by default on the initial start.

Webmentions

Have you written a response to this post? Let me know the URL:

Below you can find the interactions that this page has had using WebMention.

Web Browser telemetry - sizeof(cat)

sizeof.cat | 32 days ago

Against 3X Speed - David Perell

Mike boastfully reads 100 books per year. He listens to audiobooks at 3x speed whenever he drives and swears he can remember it all. His browser has a plugin that lets him speed up YouTube videos, and for a while, he listened to podcasts on a special app because of its unique “Smart Speed” feature. All around, his strategy for learning is simple: shove as much information into the mind as possible.

He’s one of those white-collar workers who’s always on the brink of quitting his job. He listens to podcasts while making pivot tables in Excel and stopped taking the subway to work because the train noise made it impossible to hear the audiobook narrators. During coffee breaks, he avoids conversation because he doesn’t like living life at “1x speed.” With a sped-up voice in his ear instead, he looks out of the 32nd floor of his company’s office and dreams of the day when he can finally quit his job and have the time to consume even more information.

200 books per year, he hopes.

Despite all the information he’s sped through his ear, he’s never actually built anything. “Someday,” he insists.

“Right now, I’m still learning.”

I’m not constructing a strawman here. I know many people like Mike, and I used to be just like him.

Mike is so busy preparing for the future that he never steps into it. The satisfaction of binge consumption brings instant gratification, so why try anything else? The problem is that shoving information into your mind can create the illusion of knowledge, especially when you rush it. True learning requires contemplation. And implementation. And a commitment to reflecting on great ideas over and over again.

Measuring your learning with a scorecard of consumption is anxiety-inducing. If you ground your identity in how much you’ve read, you’re always going to feel like a fraud because you’ll never be able to check every book off your want-to-read list. Plus, you can’t take action if you think you need to know everything about a topic before you step into the arena.

Mike’s Problem

Listening to audiobooks at 3x speed is born out of a flawed model of learning — and it’s the same one that underpins our modern education system. The assumption is that people can acquire knowledge as if it’s a substance they can pour into their minds. I call it the Water in a Cup method—and anybody with a lick of common sense or the initiative to read a few research papers will see how misguided it is.

What’s happened with Mike is this: he’s taken a flawed assumption of how learning happens, which he picked up in grade school, and without questioning it, brought it into his adult life. Only now, he’s turned it up a notch. Faster is better, so he no longer consumes ideas at human speed. Instead, he figures that if he can pour information into his mind 3x faster, he’ll learn 3x faster than he did growing up.

To which, Mike replies: “I mean, obviously — the math checks out, bro.”

The core assumption is that delivering more information is the best way to help people learn more. In school, most of that information is delivered through lectures, which focus on cleanly packaged ideas that are, ideally, easy to memorize and regurgitate. But thinking you know something because you can repeat what the teacher said is like calling yourself a chef because you’re good at following Blue Apron recipes. Only when you remove the scaffolding of sliced carrots and pre-packaged portions do you confront the limits of your knowledge.

Reflecting on his classroom experience, the philosopher Mortimer Adler once said: “A lecture has been well described as the process whereby the notes of the teacher become the notes of the student without passing through the mind of either.”

In accordance with Adler’s assertion, two researchers from Indiana University studied the ebbs and flows of students’ focus during a typical class period. They found that attention spans started to lapse after 10-18 minutes, no matter how good the teacher or how compelling the subject matter. After that, students’ attention would eventually return, but in briefer intervals each time. By the end of class, students could only stay focused for 3-4 minutes. Though this finding hasn’t been refuted, the implication of it — that long lectures aren’t an effective way to teach students — has been ignored by an educational establishment which still relies on lectures.

If we embraced the benefits of active learning, our classrooms would look nothing like they do today. The average classroom is set up for passive listening. It’s geared towards consuming knowledge, not integrating it. Desks are lined up in punitive rows that are designed to limit interaction between students. Subliminally, they say: “Shut up and listen to your teacher.”

In my experience, lectures are most effective when they’re only a component of the classroom experience. In fact, two of the sharpest 15-year-olds I know, who attend one of the best K-12 schools in the country, haven’t had a formal teacher since the 4th grade. My favorite college professor, Mrs. Cahill, saw how classrooms limited the integration of knowledge and rebelled against their architecture. We rearranged the entire classroom at the beginning of every class. That way, we could have the kinds of small group conversations that traditional classroom arrangements prevent. Her class was also less about consumption and more about conversation. Before arriving, we were responsible for reading 10 pages and writing a 750-word reflection on them. Class always began with small group conversations, in which we discussed the reading from the night before. By ungrouping and regrouping into different groups, we gained different perspectives on the reading.

In other classes, I slouched in my chair and watched the seconds tick for 90 minutes. In Professor Cahill’s, we were always on the move. Time was precious because each conversation was so short. Only at the end of class, when we came together for a group discussion and a short lecture to fill in our gaps in understanding, did we do anything that resembled a traditional approach. I may not have consumed the most knowledge in Professor Cahill’s class, but I certainly retained the most.

Why do classrooms revolve around the water in a cup method?

Because the water in a cup method is easy to deliver and scale.

Schools assume that learning is inevitable if students read enough books and spend enough time in the classroom. Systems are easy to scale when they use this reductive, cookie-cutter mindset. That’s why dates and key terms are bolded in textbooks and also the subject of exams at the end of the semester. But knowing the name of something without also understanding the context behind it isn’t knowledge. It’s trivia — and trivia is an ignorant person’s idea of what knowledge looks like.1

Differences between classrooms were much greater before the invention of textbooks. Teachers taught through lectures and sometimes even apprenticeships that students couldn’t find anywhere else. But the invention of textbooks standardized direct instruction, moved it into the classroom, and downplayed individualistic teaching styles. For the first time, authors had more legitimacy than the teachers who taught their textbooks. No longer were teachers the ultimate authorities on a subject who taught whatever they wanted. Instead, they served as informational middlemen who transferred ideas from authoritative writings to student minds. Every student could now study the same material, no matter where they lived. In tune with this post-industrial mindset, fuzzy and hard to quantify educational methods like apprenticeships and the singular teachings of local sages were overtaken by national benchmarks and one-size-fits-all curriculums.2 Individuality was removed from the system and replaced with the rote monotony of a trip to the DMV. Like the factories they aimed to prepare students for, schools acted like a conveyor belt where every pupil moved at the same speed.

A lecture-based class is predictable because you can prepare what you’re going to say in advance. As long as students remain in their seats, the teacher will feel like they’ve accomplished something, even if the students are just passing notes or picking their nose the entire time. Projects or Socratic discussions aren’t so simple. Since anything can happen when the students are in charge, the system doesn’t have the copy & paste repetition that industrial scale systems demand. Plus, the establishment says, if you’re a teacher, isn’t it your job to deliver information? No. Your job is to make sure they learn. That process is accelerated when you move beyond the “Water in a Cup” theory of learning.

The Alternative

The smartest people I’ve met reject the “Water in a Cup” theory. They focus less on consuming as much information as possible and more on cultivating the deepest possible understanding of the ideas that resonate with them most.

Their neverending springtime harvest reminds me of the way my approach to travel changed after some trial and error. I once went on a month-long Euro Trip during which I tried to visit as many cities as possible. In four weeks, I visited Amsterdam, Berlin, Prague, Vienna, Budapest, Interlaken, Innsbruck, and Salzburg. My goal of visiting as many cities as possible was the “Water in a Cup” theory applied to travel. Though I got to survey these cities, my international game of hopscotch made it hard to cultivate meaningful experiences. By the time I’d build an emotional attachment to a city or somebody who lived there, it was time to pack my bags. Upon returning home, I vowed to avoid trips like that in the future. Instead, I vowed to spend more time in the cities I visited.

The meaningful parts of a culture, like books, only show up when you give them time. They hide parts of their personality and only reveal themselves once you stop, slow down, and commit to them for a little while.

I noticed this recently when I spent a month in Oaxaca, a small town in southern Mexico. This time, I tried to behave less like a tourist. Only after a few weeks did I discover the kinds of local rituals that fly-in tourists rarely participate in. Most memorably, a local recommended a Temazcal ceremony—we sat in a clay sweat lodge for 75 minutes where, in pitch black, we listened a shamanic lectures and rubbed local foods like mezcal and cacao and oranges and some kind of THC-heavy substance all over our bodies. Each day also brought us closer to a woman named Petra, who prepared a new Oaxacan dish for us every day and explained the history behind each one. On my flight home, I realized that places can be as complicated as people are. Though you can pick up the gist pretty fast, relationships need time to blossom. Just when you think you know them, they’ll reveal something that adds a new dimension to their being. In travel and learning alike, you can’t rush your way to a relationship.

In school, writing essays was the closest I got to that deeper experience. If going 3x speed is like country-hopping, writing an essay is like being an expat. When you write your essay, you’re effectively saying: “This is my intellectual home for the next little while.” It’s the opposite of the intellectual nomading that’s become so popular. To that end, schools are right to see essay writing as one of the most effective methods of learning. Syllabuses, too, are useful because they force you to focus on a single topic for longer than you naturally would.

Come to think of it, writing and careful reading are like spaced repetition for the mind.

Spaced Repetition

The research on spaced repetition shows that listening to audiobooks at 3x speed is a terrible way to retain information. In fact, it’s the opposite of what you should do. If you want to retain information, you should review the stuff you’ve already read.

Spaced repetition yields exponential benefits for increased effort. So, at least at first, you gain more from a piece of information every time you return to it. Your ability to remember something improves every time you review it, and the more you do it, the less time it takes to do so. Taken all together, racing through ideas at 3x speed is precisely the opposite of what the research says you should do.

At the risk of oversimplification, humans have two kinds of memory: short-term and long-term. When you read books at 3x speed, you make it hard for your mind to transfer knowledge to long-term memory, which is more stable and lasting than short-term memory. Though brain scientists are still learning how this process of consolidation happens, researchers like Eric Kandel have written: “For a memory to persist, the incoming information must be thoroughly and deeply processed.”

The science of memory reveals why writing is such an effective way to learn. Putting ideas into your own words forces you to internalize them. Rather than memorizing disparate pieces of information, writing helps you deduce their logic and bring them together into a coherent whole. Moreover, the process of brainstorming, typing, and editing our words acts like spaced repetition. Taken all together, a systematic implementation of spaced repetition may be the lowest hanging fruit in education right now — and it’s the opposite of the “Water in a Cup” method of learning.

But if spaced repetition is so effective, why don’t people embrace it?

It’s boring. Since humans crave novelty, returning to the same ideas can be a painful experience. And besides, all the information available to us on the Internet makes us feel insecure about the thing we haven’t read, which heightens our instinct to race through ideas. Though I realize the effectiveness of spaced repetition, I have no interest in returning to the same flashcards over and over again, maybe because it reminds me so much of school.

Our intuitions for effective learning sometimes deceive us. Just because you feel like you’re learning something doesn’t mean you actually are. One study found that active learning makes students think they’re learning less even when they’re actually learning more. That’s one reason why, even though they’re less effective, lectures have persisted for so long.

Learning is most effective when we implement spaced repetition, sometimes without even realizing it. The benefits don’t necessarily come from seeing the same exact piece of information multiple times. They can also come from returning to similar states of consciousness over time. Like writing, direct experience makes that inevitable.

Direct Experience

There’s a whole level of knowledge, most of which is hard to define, that only reveals itself once you step outside the classroom and actually do the thing. I’ve always liked the idea that in theory, there’s no difference between theory and practice. But in practice, there is. That’s why, even when it’s logically sound, academic knowledge often falls short when people try to implement it.

If you want to learn, you can’t just take things in. You have to figure things out for yourself. It’s like riding a bike. The classroom can give you instructions and training wheels, but at the end of the day, you’re only going to learn by putting your butt on the seat, pushing the pedals yourself, and falling off a few times before you eventually learn. Otherwise, you’ll be like the archetypal academic who’s read every book on their subject of expertise but doesn’t have the three-dimensional knowledge that comes with taking real-world action.

Language learning provides another example. You can pick up trivial knowledge by banging your head against a classroom wall for 10 years. Or, you can immerse yourself for a year and walk away fluent. Conversations with native speakers are far more effective than learning in a classroom.

Lectures shouldn’t be the main event. Same goes for anything that resembles a lecture, like an audiobook. When it comes to language learning, you aren’t fluent when your grammar is perfect. You’re fluent when you can joke around using slang and other colloquialisms. Usually, that means you’re no longer translating words from your native language before you say them. You’re not thinking about grammar or pronunciation either. Everything flows together. The vast majority of language speakers can only reach this kind of fluency once they’ve immersed themselves in the day-to-day realities of a foreign culture.

An Alternative to 3x Speed

Look, I’m not saying Mike should stop listening to audiobooks. They’re a savior because he lives an active lifestyle and struggles to sit down for long periods of time. Though he originally listened to audiobooks at their standard speed, he caved to the urge to listen to each one faster once his stack of unread books grew to an anxiety-inducing length. First, it was 1.25x speed. Then 1.5. And now, 3x.

But if he’s doing 3x speed now, where does it end? Should he crank it up to 5x? How about 7x? At some point, the quest for speed breaks.

Audiobook sprinters sometimes defend their habit by citing the research showing that some languages are spoken faster than others. Maybe, just maybe, that implies that we can crank up audiobook speeds without harming recall. But even though some languages are spoken faster than others in terms of syllables per second, the amount of information communicated per second is roughly equal, no matter what language you’re speaking. There’s a tradeoff between complexity and speed. The higher the information density of a language, the slower it is spoken.3

It’s okay to not know everything. The world rewards people who develop expertise in a specific subject. When that expertise is unique, it’s developed through direct experience and deliberate reflection. Mike should study the meta-practice of learning itself and embrace a model of learning that respects the limitations of the human mind. The water in a cup theory is a false, robotic assumption that was instilled in us, wrongly, by an archaic education system.

Newsflash: Mike is a human, not a computer. He needs time to synthesize what he reads and transfer knowledge from short-term to long-term memory. He should think more strategically about what he wants to learn and why. By learning at a more human speed, he can spend more time integrating knowledge. Racing through audiobooks isn’t helping Mike as fast as he thinks. He’ll serve himself far better by slowing down and embracing the triad of writing, spaced repetition, and direct experience.

Thanks to Ellen Fishbein for helping me write this essay.

Cover Photo by Clem Onojeghuo on Unsplash

Against 3X Speed - David Perell