Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Show HN: I Built 0xFast Stream – 100x Faster Ethereum Block Downloads (manishrjain.com)
13 points by mrjn on Sept 22, 2022 | hide | past | favorite | 8 comments
Getting web3 data out of Ethereum nodes or node services is excruciatingly slow (at least days if not weeks). I had wanted a way to just download the blocks quickly since I started exploring the web3 space. Took me a bunch of trial and errors to get to a way where I can parse all the blocks since genesis within 12 hours. But, instead of moving on to the next thing, I put this service together over the last few days, so everyone can benefit.


wow can't believe just yesterday I thought how cool it would be to have that - and today you made it! :D

downloading...

btw did you verify (PoW) blocks using the nonce? just to be on the safe side :)


also if you could use binary encoding rather than json you could probably cut the size by half.

I already started download, but perhaps it's worth noting in your website to use "--compressed" flag like so:

curl --compressed -H "Accept-Encoding: gzip" https://stream.0xfast.com/beta/eth-mainnet/1-latest


Oh, --compressed is is a great point, thanks. Updating the blog.

I wanted to stick to JSON -- it's just the easiest for everyone to work with. And it compresses well, so all good.


Glad it's useful! :-)

> btw did you verify (PoW) blocks using the nonce?

I've validated the data and checksummed while exporting from Geth. But, perhaps can validate data when serving it as well. Any pointers / articles on the best practices here?


Are you exposing full node or archive node? Because Archive node is several terabytes :)


It doesn't have the states yet. So, equivalent of full nodes -- block, txns, receipts, logs (no traces, states).

But, could be extended to include those, if needed.


Curious how you managed compute cost. Are you running a node locally?


Things are cheaper when you put together your own hardware. For example: https://manishrjain.com/building-a-2-5-gbps-load-balancer-se...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: