By the way, anyone have experience using emacs to analyse and visualise data (think spreadsheets and charts)? I’d really like to be able to use it to view any sort of data I happen to have.
I use the `vnlog` and `feedgnuplot` shell tools HEAVILY to do data analysis and visualization. In emacs, these work well in shell-mode or in any buffer with `shell-command-on-region` (M-|). Not strictly emacs, but works great.
An uninformed question: If the network is fully composed of 1x1 convolutions, doesn’t that mean no information mixing between pixels occur? Would that not imply that each pixel is independent of each other? How can that not lead to incoherent results?
This apparently doesn't apply here, but in fact, pixels can be generated independently of each other. There are architectures where you can generate an arbitrary pixel or element of the image without generating the others; they are just implicit. See NeRFs or 'single-pixel GANs' or MAEs: eg https://arxiv.org/abs/2003.08934https://arxiv.org/abs/2011.13775https://arxiv.org/abs/2401.14391
Why is this possible? I tend to think of it as reflecting the ability to 'memorize' all possible data, and the independent generation is just when you 'remember' a specific part of a memory. The latent space is a Platonic object which doesn't change, so why should your generative process for materializing any specific point in the latent space have to? It's not surprising if you could generate arbitrary points from a function like 'y = mx + b' without generating every other point, right? It's just an atemporal mathematical object. Similarly with 'generating images from a random seed'. They too are just (complicated) functions mapping one number to another number.
(You might wonder if this is limited to images? It is not. In fact, you can generate even natural language like this to some degree: https://github.com/ethan-w-roland/AUNN based on my proposal for taking the 'independent generation' idea to a pathological extreme: https://gwern.net/aunn )
In DDN, 1×1 convolutions are used only in the output layers of the Discrete Distribution Layer (DDL).
The NN blocks between DDLs, which supply the fundamental computational power and parameter count, adopt standard 3×3 convolutions.
What are some more information on the state of the art in runtime adaptation? I confess I do not feel like I possess such a thing from the databases I regularly use.
Adaptation sounds very compelling; if instead of emitting a plan based on a cardinality estimate, we emit a plan and a range of reasonable intermediate cardinalities together with expected time, and interrupt the original plan when the expectations are exceeded by an order of magnitude, and perform alternative plans based on newly gathered physical information, it sounds like it would be greatly beneficial. Are there concrete reasons that this has not been done (e.g. cost, complexity)?
Adaptive planning also applies to adapting the degree of query parallelism and automatic banning of query plans that take too much time.
Commercial databases have much bigger teams than Postgres does and have a lot more optimizations as a consequence. This includes optimizations that aren't in scope for an open source DB to begin with. For example, Oracle uses the Apple strategy for decades. A lot of Oracle DBs are sold as hardware not software, shipped as a full cluster rack or set of racks to the customer. The gear uses a dedicated inter-node 100Gb/sec ethernet within the cluster to give horizontal scalability of joins and other complex read/write loads. Queries and results go in/out over a separate frontend network. There are customizations up and down the stack from kernel, firmware, device drivers, database nodes and the client drivers. Nodes directly read/write each other's memory using RDMA to fetch the latest version of rows from each other or inform each other about new transactions.
But yeah you can snatch traversal statistics from your index accesses as you're executing and if those hit planner-provided thresholds you go do what the planner said to do/check in case of that specific trigger, then feed your statistics and the extra queried info to the planner to get your plan adjusted.
Doing this without creating unpredictable performance bugs let alone any correctness bugs is going to be hard, though.
Even more when you consider it also has to be fast because otherwise it wouldn't be worth using.
You may want to check WCOJ technology where you can run a multi-way-join with cardinalities being of a "could well be >10, don't know and even if it'll vary across data/keys"-nature:
1. You index all data as needed for perfect index lookup joins on ANY non-stupid join ordering that query could be run as.
2. You prepare separate indices that tell you the number of values your single-key lookup join would return, for each key, for each such possible index lookup join included in (1).
3. For querying you start with a key or key range that could be fed to an index from (1) and then for each tuple at each step between individual joins you ask the candidate indices of (2) which one would thus be the cheapest to expand with, use the selected index of (1) (as told by the fine-grained cardinality counts of (2)) to expand your tuple, then choose the next relation to expand with.
If you ever have a choice where you know the cardinality of a key in (1) is always 0 or 1, you don't need an index of (2) for that index of (1) and you will always greedily apply those, preferring those that have a higher chance of killing the currently evaluating partial tuple from propagation through further indices.
That alone btw. is sufficient to get the worst case optimal runtime of triangle listing/counting in arbitrary edge lists of O(n^(3/2)) (incidentally equal to the maximum possible number of triangles) instead of the naive/classic O(n^2). n being the number of edges.
Constants here for the triangle case are on the order of 10~50 depending on how hard you want to argue vectorizing and using flat array "columnar" data structures instead of in-memory structures that allow incremental updating without brute force rebuilding from scratch at every change. E.g. supporting adding (but not deleting) edges and efficiently getting updated results is something that doesn't fit the most aggressive performance optimizations that would give the ~50 constant factor vs. a "mere" ~10 when you want to support incremental updates.
(At constant=10 break even would be at n=100; at constant=50 break-even would be at n=2500; IMO it's therefore quite relevant in many practical situations with how quick those asymptotes hit results/benefits.)
When things aren’t ticking perfectly for Synology their software can be kind of weird. Sometimes after power failures, some disks get corrupted and… you simply can’t log in to the Synology UI during this time (unless you “synobootseq --set-boot-done”, why, of course) for an unspecified number of hours.
Their custom software has its quirks (eg scp doesn’t work unless you apply the -O flag, for “security” reasons), and the quirks change sometimes after updates.
I’ve once attempted to implement a solution where ids are generated by UUIDv5 from a certain owner and the relationship of the new item to the owner; that way, users cannot generate arbitrary ids but can still predict ahead of time their new ids to ease optimistic behaviour.
For those who are not aware, knowing how to count the number of structures is (nearly) the same as knowing how to encode said structure as an integer (space-optimally) (the naive general algorithm would not have good time complexity); to convert to integer, simply define some ordering on said structures, enumerate until you find your structure, the index of the structure is the integer encoding; conversely, to decode, simply list all structures and pick the structure with the index of the integer.
> simply list all structures and pick the structure with the index of the integer
This sounds like to decode a single item you have to do work proportional to the cardinality of the set? Optimal space efficiency comes at a high computational overhead?
Yeah, its a theoretical general connection. Once you’ve pinned down the specific structure and know how many structures to skip by virtue of knowing how to count them, it becomes a somewhat practical algorithm.
A somewhat more “complete” solution that doesn’t give you hints (thus doesn’t rely on the plugin support for all of vim’s vast functionalities), but conditions your instincts to get better: increase the latency of my whole terminal (c.f. https://unix.stackexchange.com/questions/778196/how-to-add-d...) (also see the comment) by running my terminal session on a ssh session into my own machine through a ProxyCommand of the command delay.
A very exaggerated example to use exp time… Whatever your constant, for exp time, it’s going to run for times longer than the life of the universe on probably any n > 10… Since you know, log time is practically constant
Batching and throttling. If you’re doing work with a loose deadline don’t make it outcompete work with a tight deadline. That’s a self-DOS. Make the batch no more than 1/10th to 1/8th of the overall traffic and things go more smoothly.
Halting (problem) is undecidable. Undecidability is not just the halting problem. Whether you go to the park on a given day is undecidable (assuming you have free will, and all such assumptions), but that you can be an oracle for this does not solve the halting problem.
> Whether you go to the park on a given day is undecidable
I feel like that is not really undecidable in the same way the halting problem is. The result does not depend on the input. It'd be like saying checking primality is undecidable if you dont know what number you're supposed to check.
