The platform is quite open, so you can build whichever tests you want. Personally I run a dockerized ooni via scheduled GitHub actions, which shows blocks impacting GitHub runners. You should be able to see logs from that here, which shows three blocked websites https://github.com/robalexdev/ooni-unattended-action/actions...
For some of that, you don't need to crawl. Wikipedia offers database dumps which you can download in one go. Lots of programming docs are managed in repos, so you can clone the repo instead. Even stackoverflow seems to have a snapshot dump (https://archive.org/details/stackexchange).
I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3.
I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...
For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.
Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs.
So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive.
It didn't work, or the gains were so marginal that I ended up scrapping the approach.
So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.
When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog.
Imagine how fast me and the dog ran back 30 minutes later when I realized how cooked I was. I only had 10 GB left on my disks after I downloaded that 20 GB file. This decompressed file would have blown the machine up. I was terrified to find a frozen system with no storage space left.
Instead, the process finished and `df -h` reported 8 GB of the free space left. Files were decompressed. I could `less` them! That made no sense! Only many many minutes later I finally figured out to run a `zfs get compressratio` command which showed ZFS successfully and transparently recompressed everything on the fly. That was too impressive for that teenager and he never switched to a different file system.
Solaris has had so many cool features, like ZFS or doors. What I liked about ZFS is you coul make snapshot, which is basically the solution to how to treat data files a single, cheap to access unit, yet still use standard apis for file management, great for containerizing apps, making copies for experiments, or shipping stuff. Node.js just received this as a bespoke, app-level feature. But these things are too many to count, and make a ton of sense if you know how filesystems actually work. useful Also copy-on-write, temp overlays.
Pretrained dictionaries have never been intended to help with book sized or bigger compression. zstd automatically learns the most efficient dictionary it can within a few kilobytes. Pretrained dictionaries are only useful when you're independently compressing very small records.
Note that the dictionary options are only needed to improve compression ratios when compressing lots of small messages. If you have a bigger file (eg a tar file of Usenet messages) the regular Zstd compression will build a good dictionary without additional options.
The thing zstd got really right is fast decompression. For write-once read-never data like backups lzma (aka xz/7zip/lzip) is great. But it takes forever to decompress. On zstd I can get good compression while decompressing the file only marginally slower than reading the uncompressed file from SSD
Writing your files directly into a compressed stream and decompressing on the fly has become almost a standard workflow for any files I'm going to read and write sequentially anyways. No need for the data to ever exist uncompressed on the file system. Previous formats never did that for me because they either had too much overhead or too little gain, often both
But zstd is super tunable. Where gzip gives you compression levels from 1 to 9, zstd gives you up to 22 for ultra compression and negative compression levels for ultra fast. The ultra fast options so fast that they are great as a substitute for memcpy if your CPU is already waiting for other things, like DRAM.
No it’s not. The pace of improvement of CPU compute speed is far greater than that of DRAM throughput. And in fact compression algorithms geared towards speed aims to outperform memcpy (on suitable machines).
zstd has a built-in benchmark mode to compare different compression levels, e.g. `zstd -b1 -e9 [FILE]` to test levels 1 to 9 (try up to 22 if you have enough spare time)
Duckdb supports loading and saving to zstd for all it's base loading/saving formats csv/tsv/json/jsonlines, but, for good or bad, those are solid compression.
Under most r/w workloads, using parquet/lance/vortex/native-duckdb, with their built-in columnar compression will result in more performance AND space savings. Non-solid compression. Then, the query engine can push down your query predicate to a column row group level, instead of forcing it to decompress the entire dataset to operate.
Practical example: duckdb has syntax - https://duckdb.org/docs/lts/data/multiple_files/overview - to glob multiple files at once, but that really only works if you're applying push down query predicates instead of re-decompressing your entire data set per SELECT. I would say for most dataset, even 20%+ size is worth not having to decompress (or even download!) the entire dataset, to figure out if something fits the predicate.
After all, if you have to download and decompress the dataset back again to operate, then the "space savings" are gone.
It isn't really the go-to compression format, because it isn't ubiquitous like gzip and zip, there are a variety of compression tools out there for different purposes, and there is image/audio/video compression. There is also specialized compression like what git does with its rolling hashes. I think of it as there not being a go-to compression format.
A go-to thing means it's a sensible default choice and has no little to no downsides (versus not using compression), it doesn't mean it's the best for everything.
Until now the go-to has been DEFLATE (gzip and zip) but zstd is definitely competing against it because it is better in almost every way.
I think DMARC works well because email tends to blindly trust DNS (opportunistic encryption). On the web we expect authenticated TLS, often strictly enforced (organization policy, HSTS). So it would feel weird if a website changes how it handles HTTPS cookies based on an insecure DNS record, perhaps delivered by the resolver on an untrustworthy WiFi router.
Specifically, if I register subdomain attack.co.uk and set up a malicious WiFi router, I trick some *.co.uk cookies to get set on co.uk and then steal them from attack.co.uk by tampering with the (proposed) SVCB record.
I think the signal needs to be secure, which means DNSSEC. Adding a hard requirement for DNSSEC validation in all web browsers is a huge change from where we are now.
And essentially boils it down to ‘either the client implements wire-security to a known dns server using DoH or DoT, implements dnssec to verify the untrusted response as legitimate, or the client risks being mitm’d to attacker addresses’. They ultimately sidestepped the problem by structuring it to be hints rather than guarantees and thus allowing DNSSEC to be optional, and so as of today, it’s definitely not sufficient to implement this.
I think that adding a CORS rejection to DNS — declaring subdomains independent of a TLD, that is — does not require DNSSEC, so long as clients adhere to the steps to prohibit attacker interference described. But it still asks a great deal of DNS that I’m unsure is possible today, not just in DNSSEC but in ripple-subward records that somehow tie into client responses.
More likely, I assume browsers will simply permanently end all service to the concept of subdomains at all; no cookie sharing across domains at all, no inherent cross-origin just because tld and www.tld share a few characters, etc. rather than either depending on the PSL or having to implement strange and complex DNS anything. Admins will throw their hands up about it, but the net is no longer a place where control of a TLD defines the trust of its subordinates, so it’s certainly time to rip that bandaid off if they haven’t yet.
There are limits to how accurate you can make this. Zones like .xyz renew at 18%, so you'd expect 0.2% of those domains to expire without renewal each day. The tranco list is based on a 30 day look-back and I've seen a small percent of those domains lack name servers, even for the latest list.
Similarly, domains can be registered since the last time you downloaded your lists. So you never have a complete or accurate list.
I think it's perfectly reasonable to suggest names that have recently expired or domains that could have been recently registered. The alternative requires an NS lookup.
For blog posts, I'd look to RSS instead. That's where that content is traditionally published. Instead of SELECTing from bluesky's index, you can use OPML subscription lists. There are a bunch of places that curate feed lists, so it's significantly less likely to face API death like twitter did.
I run another variant which collects OPML blogrolls via crawling, so you can find out who else likes your favorite blog and what else they recommend. Here's the page for Simon Willison's blog (https://blogroll-network.alexsci.com/discover/feed-a34ee2a88...). Thinking of RSS and blogrolls as a network feels much more resilient than blueskys Jetstream api endpoint.
> Thinking of RSS and blogrolls as a network feels much more resilient than blueskys Jetstream api endpoint.
Nothing to do with Bluesky services. There are many independent firehoses and relays. Here's a stream of standard.site blog posts coming in over a firehose hosted in Chennai. Every single one. No curator between me and the posts, and no work by me to crawl the whole network for them. https://pdsls.dev/jetstream?instance=wss%3A%2F%2Fchennai.fir...
Edit: ok there is an aggregator, the relay is scraping all of the PDSs out there to build the event stream. Notably this is not possible with RSS, where you need to build a large index with knowledge. PDSs request relays to crawl and that's that.
> Here's a stream of standard.site blog posts coming in over a firehose hosted in Chennai. Every single one.
This stream lacks any blog posts that haven't been specifically published to atproto. Conversely, I see RSS feeds for the content in the stream, like https://bsky.app/profile/did:plc:byf7jvh3yvhffackiumpddtf/rs... (if RSS feeds exists for every profiles then I'd argue that atproto is a strict subset of RSS, but I'm not certain how that feature works). The stream may give you every blog post that was published to atproto, but that's already a small subset of long-form content on the web.
> Notably this is not possible with RSS,
The web supports blog discovery so well that people forget it exists. A simple Google search can find a very large amount of content, much available over RSS/Atom. Many web-based feed readers index the subscriptions across all their users to help with discovery. Here's the Feedland firehose, for example: https://feedland.com/?everything=true
Oh the opml blogroll map is pretty exciting! I occasionally try to browse around for these manually but they can be hard to explore for so having a big list is a big deal. I usually just do searches for them + use a scour.ing interest to find more. I'm excited to look at it more this weekend when I'm not so tired.
One thing I'd love to see with rss / ompl sharing sites is more ease of exploration. It feels a bit clunky browsing many of these feed sharing sites because you need to evaluate each feed by manually clicking through each one etc or worse you can only import the whole opml at once flooding your feeds. For example I've seen some sites with feeds directly showing posts as they come in for each persons feed list(s) so you can see what that batch of rss feeds looks like in action without clicking around much. There are some sites doing great work getting people to share their lists but I think there is room for qol improvements.
One site I've been liking that has feeds for each list and is trying to reduce friction on sharing is blogflock.com. You can also follow other peoples lists directly on the site too so they show up in your main feed. I believe I've also seen some people self host their blogroll on their personal sites with similar feeds but I'm having trouble finding that software atm.
Absolutely. One good thing going for this approach is that anyone can grab the OPML export (the URL is stable) and build their own frontend, I'd love to see more.
Could be a fun weekend project for frontend folks.
You can build your own lists and share them using ooni run: https://run.ooni.org/
reply