Data Scientology

Decoupled Descent: Enforcing Exact Train-Test Error Tracking Via AMP Onsager Corrections R Link: https://arxiv.org/pdf/2604.27883 Hi, Most of use are familiar with the headache of training a neural network using gradient descent where the training error may go to zero but the test error may stay the same as initialization or even increases. My paper treats this phenomena as a consequence of data reuse bias and can be isolated by studying full batch gradient descent on a set of stylize Gaussian mixture models. I turns out that this fundamental issue can be avoided using some clever tricks from high-dimensional statistical theory, specifically approximate message passing (which is beyond the scope of this post but I would be happy to explain more). By doing so I created a training method called Decoupled Descent (DD) which generates a certificate that the training error of the network will asymptotically equal the testing error at each parameter iterate. I think this method gives a cool way to approach how to train networks and I was hoping to get y'alls input on it. It opens up some nice ideas for optimal stopping or hyperparameter tuning and future directions of pushing to something like SGD or more general models. I have attached the train-test curves on a simple model fitting problem to compare the performance of GD with with DD (my algorithm) to give a high-level idea of what the method can guarantee. I stress this is a theory paper so there is a long way to go to get to very large models but I think it is a good first step. 100 simulations of a simple high dimensional XOR model for a bespoke two layer network. Left is training with GD, right its training with my method. The colored bands are 25% to 75% quantile. Happy to answer whatever questions people have, I plan on writing a PyTorch compatible package for this training method one day so any feature suggestions would be welcome as well. https://redd.it/1vlu1se @datascientology

Run SAM3 and RTMPose over 1950s-era factory footage. No fine-tuning. It just works https://redd.it/1vhp0h6 @datascientology

I Compressed Bad Apple into a 3MB Neural Network [P] https://redd.it/1vfrco1 @datascientology

Is it too late regain some coherence in the ML research space in our life time? D Was just looking at the list of preprints on Arxiv cs.LG https://arxiv.org/list/cs.LG/recent?skip=0&show=500 Everyday 100 - 400 new machine learning papers gets uploaded on this server. Looking at this unending list of preprints is as if you stepped into a crowded room, like the stock trading floor on wall st. in the 1980s. Everyone is shouting over each other. Nobody is talking to each other. Everyone's trying to prove something, to someone, to themselves, to build some credentials in the ML/AI space to meet those job requirements, or dying to get their truth out. Every title contains some new terminology invented by the authors that feels not worth the effort in keeping it in your working memory. Burn-out by endless novelty. Frontier research are now corporate trade secrets that politicians and military are watching closely. Research papers are ir/unreproducible he-said-she-saids. Marketing material are research paper and vice versa. Extremely major breakthroughs are announced via tweets, whereas extremely minor results are unannounced via journals. Everything feels simultaneously mostly true and possibly false (because nobody is seriously checking). Nobody knows what's going on, and people who knows what's going on has a non-disclosure clause in their job contract. Is the theory of generalization that we learned in school true or false? It feels false, why hasn't there been any retractions? Many questions like these. Is it too late to regain some coherence in this field?? https://redd.it/1ve7chh @datascientology

Beginner here: My pothole detection model mistakes the roadside for potholes. https://redd.it/1v90113 @datascientology

30+ officially free AI/ML books, all in one curated repo https://redd.it/1v7cvqr @datascientology

Are there some textbooks that take a primarily engineering approach to machine learning (as opposed to a "scientific" approach)? D As someone who studied stats undergrad and industrial engineering operations research grad, and who thinks about the practical business of ML components in software.... I get lost and a bit hopeless when I think about how to make useful software out of ML models in a reasonable amount of time, and in the current business environment. And when I look at the businesses where I have worked that have mountains of middle management running tiny bits of the ML model lifecycle (think feature extraction, data ingestion and integration, training infra, hosting infra, more hosting infra, applied science)... that only makes my head hurt even more. How do you go about making practical software out of ML components? Edit: I should mention that I mean from scratch ML components, not just a call to a third party hosted tool. https://redd.it/1v16l6a @datascientology

SenseNova-Vision is open-sourced: handle every CV task as unified multimodal generation https://redd.it/1uyorje @datascientology

I reviewed that boy. https://redd.it/1uwsmbo @datascientology

Prompt-engineering paper accepted to ICML R "Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity" This paper was accepted to ICML this year. Its main idea is a very simple prompt-engineering trick: "changing the prompt this way led to more diverse sampling". Naturally, it is difficult to provide a rigorous theoretical analysis for something like this. Even if it works, I’m not sure this kind of prompt engineering belongs at a top-tier machine learning conference. Some people seems to call this kind of work “modern machine learning”, but I think it should be categorized as less technical venues. How do you think? Am I being too rigid? https://redd.it/1uv1xb3 @datascientology

Hyperparameter tuning approach question R I am doing some work with cell type classification, where I have 4.3 million cells and 512 features (condensed embeddings from the encoder of a transformer). The broader goal is to implement a contextual bandit for augmenting the training set of the dataset, as it is currently imbalanced, and rare cell type classification is poor when I tried a baseline logistic regression classifier. Dataset: Feature matrix shape: (4290471, 512) Labels shape: (4290471,) Class distribution: T cell 1966941 DC 858451 NK cell 561904 Monocyte 411170 B cell 375882 Platelet 54576 Progenitor cell 24689 ILC 24254 Erythrocyte 12604 I didn't do any hyperparameter tuning for the LR classifier, but I want to try other ML models (LightGBM, XGBoost, SVM) However, I face a bottleneck with hyperparameter tuning. I want to do 80/10/10 train/validate/test split, but the training set is so large and takes a long time even on H100. What are some solutions to this? I tried optuna but still very long for each hyperparameter trial. I then tried optuna but instead of using the full 80% for training each time, only 15% of the 80% is used (subsampling from the training set). I'm not sure if this is robust or not. I also couldn't really find anything in the literature. Anyone been in a similar situation? https://redd.it/1usa46w @datascientology

TorchJD: Training with multiple losses in PyTorch P Hi everyone! I wanted to share some recent progress on TorchJD that might be useful to the machine learning community. When training models with multiple losses (multiple tasks, constraints, auxiliary losses, regularization terms, etc.), you typically have two options: Scalarization: Various ways to combine those losses into a single loss (e.g. average them or combine them with trainable weights); then you can do gradient descent on it. Jacobian descent: Compute the Jacobian of the vector of losses (i.e. one gradient per loss), and aggregate it into an update vector that will decrease each individual loss (rather than just the average loss). There are many ways to do this aggregation step. Scalarization methods are generally cheaper in memory, but in some cases there is so much disagreement between your objectives that it's better to use a Jacobian descent method. In any case, thanks to our amazing new contributors, we've now finally implemented most existing methods of the literature from both categories into our library TorchJD, so that you can try anything in just a few line changes! Recently, TorchJD has been accepted into the PyTorch ecosystem, and we're trying to make it become the go-to library for training with multiple losses. If you'd like to help build the future of the project, come join us on Discord (link can be found in the readme of the repo). New ideas, contributions, bug reports, experiments, and any form of feedback are all welcome. We have many ideas on how to make all this even more efficient, and we will need help for that. If you want to support us, a star on GitHub also helps a lot! https://redd.it/1upzxk2 @datascientology

I trained a local AI model that generated 22,000+ novel drug-like molecules — verified against 4.6M known compounds. Dataset available. Built an 80M parameter causal transformer on consumer hardware (RTX 5070), trained on MOSES + ZINC-250k. Generated and filtered for QED ≥ 0.5, SA ≤ 4.0, MW ≤ 500. Top compound hits QED 0.947. 100% novel against MOSES, ZINC, and ChEMBL. HuggingFace: https://huggingface.co/datasets/MKEChem/mke-novel-druglike-smiles Happy to answer questions about the generation method. https://redd.it/1uojccn @datascientology

Books/Resources to improve mathematical foundations for ML research D I am a mid to late stage PhD student in ML. I've known this before, but only recently I started feeling this urgently: my mathematical foundations are shaky, because I kept "learning-things-as-I-go" when working on various problems. I likely have only a year or two left until I graduate, and before I do so, I want to really dedicate some time and focus to brush up on the fundamentals. Primarily, I want to improve my knowledge in Linear Algebra, Probability Theory, and Functional Analysis. For Lin. alg., I am looking at "Linear Algebra done right", and I think this book is sufficient for the topic, unless anyone thinks otherwise. I am not sure where to start for probability, as well as functional analysis. Rudin's books give me headaches. I instead started reading "A primer on RKHS" (https://arxiv.org/abs/1408.0952) to "dip my toe" into functional analysis. Apart from the above, I might re-read PRML book (I've only read specific chapters before), and try to finish Pat Kidger's Just-Know-Stuff list (https://kidger.site/thoughts/just-know-stuff). Thoughts? Anyone have any book/resource recommendations? Someone told me to look into "the bright side of mathematics" on YouTube, anyone ever go through the videos there? I'm aware finding good, digestible resources is less than 10% of the challenge. The difficult part is sticking through and actually reading/working through these topics, while still juggling other academic responsibilities. https://redd.it/1ulmy9g @datascientology

WIP: Currently building an app to teach (French) sign language using computer vision https://redd.it/1uk1lk7 @datascientology

A physical, working LeNet-1 (1989) built from transparent PCBs, glass and aluminium. https://redd.it/1uhr1g1 @datascientology