Publications

For an updated list please visit Google Scholar. * denotes equal contribution.

2024

1. arxiv

   

   VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images

   M. Maruf, A. Daw, K. S. Mehrab, and 19 more authors

   ArXiv, 2024

   Abs arXiv Code

   Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images.

2. arxiv

   

   What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits

   H. B. Manogaran, M. Maruf, A. Daw, and 10 more authors

   ArXiv, 2024

   Abs arXiv Code

   A grand challenge in biology is to discover evolutionary traits—features of organisms common to a group of species with a shared ancestor in the tree of life (also referred to as phylogenetic tree). With the growing availability of image repositories in biology, there is a tremendous opportunity to discover evolutionary traits directly from images in the form of a hierarchy of prototypes. However, current prototype-based methods are mostly designed to operate over a flat structure of classes and face several challenges in discovering hierarchical prototypes, including the issue of learning over-specific features at internal nodes. To overcome these challenges, we introduce the framework of Hierarchy aligned Commonality through Prototypical Networks (HComP-Net). We empirically show that HComP-Net learns prototypes that are accurate, semantically consistent, and generalizable to unseen species in comparison to baselines on birds, butterflies, and fish datasets.

3. ECCV 24

   

   Hierarchical Conditioning of Diffusion Models Using Tree-of-Life for Studying Species Evolution

   Mridul Khurana, Arka Daw, M. Maruf, and 12 more authors

   In Proceedings of ECCV, Milan, Italy, 2024

   Abs arXiv Website

   A central problem in evolutionary biology is to explore the genetic basis of evolutionary changes in the traits of organisms, such as fin structures in fish or beak colors in birds. With the growing availability of large-scale image repositories in biology and recent advances in generative modeling, there is an opportunity to study changes in evolutionary traits of species automatically from images. We introduce a novel Hierarchical Embedding (HIER-Embed) strategy to encode the evolutionary information of a species as a composition of encodings learned at every internal node in the phylogenetic tree. We use HIER-Embeddings to condition latent diffusion models to generate synthetic images of species. Further, we introduce two novel types of perturbation operations: trait masking and trait swapping, similar in spirit to gene knockout experiments, that enable us to analyze novel changes in evolutionary traits acquired at different levels of phylogeny.

4. arxiv

   

   FishVista: A Multi-Purpose Dataset for Understanding and Identification of Visual Traits from Images

   K. S. Mehrab^*, M. Maruf^*, A. Daw^*, and 14 more authors

   In ArXiv, 2024

   arXiv Code

2023

1. Vision

   

   Beyond Discriminative Regions: Saliency Maps as Alternatives to CAMs for Weakly Supervised Semantic Segmentation

   M. Maruf, Arka Daw, Amartya Dutta, and 2 more authors

   In arXiv, 2023

   Abs arXiv Code

   A central goal in deep learning is to learn compact representations of features at every layer of a neural network, which is useful for both unsupervised representation learning and structured network pruning. While there is a growing body of work in structured pruning, current state-of-the-art methods suffer from two key limitations: (i) instability during training, and (ii) need for an additional step of fine-tuning, which is resource-intensive. At the core of these limitations is the lack of a systematic approach that jointly prunes and refines weights during training in a single stage, and does not require any fine-tuning upon convergence to achieve state-of-the-art performance. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM). The key intuition behind DAM is to discriminatively prefer some of the neurons to be refined during the training process, while gradually masking out other neurons. We show that our proposed DAM approach has remarkably good performance over various applications, including dimensionality reduction, recommendation system, graph representation learning, and structured pruning for image classification. We also theoretically show that the learning objective of DAM is directly related to minimizing the L0 norm of the masking layer.

2. Ecology

   

   A FAIR and modular image-based workflow for knowledge discovery in the emerging field of imageomics

   Meghan Balk, John Bradley, M. Maruf, and 14 more authors

   Methods in Ecology and Evolution, 2023

   Website
3. AAAI 23

   

   Are Pre-trained Vision Language Models (VLMs) Decent Zero-shot Predictors in Scientific Contexts?

   M. Maruf, Arka Daw, and Anuj Karpatne

   In first workshop on Imageomics at AAAI, 2023
4. AAAI 23

   

   Conditioning Diffusion Models Using the Knowledge of Phylogeny for Understanding Species Evolution

   Mridul Khurana, Arka Daw, M. Maruf, and 1 more author

   In first workshop on Imageomics at AAAI, 2023
5. AAAI 23

   

   Phylo-GNN: Phylogeny-guided Graph Neural Network Approach for Fine-Grained Image Trait Identification

   Kazi Mehrab, Arka Daw, M. Maruf, and 1 more author

   In first workshop on Imageomics at AAAI, 2023

2022

1. Biodiversity

   

   Extracting Landmark and Trait Information from Segmented Digital Specimen Images Generated by Artificial Neural Networks

   Yasin Bakış, Bahadır Altıntaş, Xiaojun Wang, and 3 more authors

   Biodiversity Information Science and Standards, 2022

2021

1. NeurIPS 21

   

   Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)

   Jie Bu^*, Arka Daw^*, M. Maruf^*, and 1 more author

   In Advances in Neural Information Processing Systems (NeurIPS), 2021

   Abs arXiv Code

   A central goal in deep learning is to learn compact representations of features at every layer of a neural network, which is useful for both unsupervised representation learning and structured network pruning. While there is a growing body of work in structured pruning, current state-of-the-art methods suffer from two key limitations: (i) instability during training, and (ii) need for an additional step of fine-tuning, which is resource-intensive. At the core of these limitations is the lack of a systematic approach that jointly prunes and refines weights during training in a single stage, and does not require any fine-tuning upon convergence to achieve state-of-the-art performance. We present a novel single-stage structured pruning method termed DiscriminAtive Masking (DAM). The key intuition behind DAM is to discriminatively prefer some of the neurons to be refined during the training process, while gradually masking out other neurons. We show that our proposed DAM approach has remarkably good performance over various applications, including dimensionality reduction, recommendation system, graph representation learning, and structured pruning for image classification. We also theoretically show that the learning objective of DAM is directly related to minimizing the L0 norm of the masking layer.

2. KDD 21

   

   PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics

   Arka Daw^*, M. Maruf^*, and Anuj Karpatne

   In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (KDD), 2021

   Abs arXiv Code

   As applications of deep learning (DL) continue to seep into critical scientific use-cases, the importance of performing uncertainty quantification (UQ) with DL has become more pressing than ever before. In scientific applications, it is also important to inform the learning of DL models with knowledge of physics of the problem to produce physically consistent and generalized solutions. This is referred to as the emerging field of physics-informed deep learning (PIDL). We consider the problem of developing PIDL formulations that can also perform UQ. To this end, we propose a novel physics-informed GAN architecture, termed PID-GAN, where the knowledge of physics is used to inform the learning of both the generator and discriminator models, making ample use of unlabeled data instances. We show that our proposed PID-GAN framework does not suffer from imbalance of generator gradients from multiple loss terms as compared to state-of-the-art. We also empirically demonstrate the efficacy of our proposed framework on a variety of case studies involving benchmark physics-based PDEs as well as imperfect physics.

3. Biology

   Biology-guided Neural Network for Fish Trait Discovery

   M. Maruf, M. Elhamod, PK Mandke, and 1 more author

   In Integrative And Comparative Biology Workshop, 2021

   Abs

   Unlike genetic data, the phenotypes, or traits of organisms such as their visible features, are not available in databases for analysis. The lack of machine-readable trait data has slowed progress on four grand challenge problems in biology: predicting the genes that generate traits, understanding the patterns of evolution, predicting the effects of ecological change, and species identification. The BGNN project aims to leverage advances in state-of-the-art machine learning to develop a novel class of artificial neural networks, termed biology-guided neural networks (BGNNs), that can exploit the machine readable and predictive knowledge about biology that is available in the form of phylogenies and anatomy ontologies. These BGNNs are expected to automatically detect and predict traits from specimen images, with little training data. Currently the project is focusing on teleost fishes because of many high-quality data resources available (digital images, evolutionary trees, anatomy ontology). The resulting machine learning model can be generalized to other disciplines that have formally structured knowledge, and will contribute to advances in computer science by going beyond black-box learning and making important advances toward Explainable Artificial Intelligence. Image-based trait data derived from this work will enable progress in gene-phenotype mapping to novel traits and understanding patterns of evolution. It may be extended to applied areas, such as agriculture or the biomedical domain. This convergent research will accelerate scientific discovery across the biological sciences and computer science by harnessing the data revolution in conjunction with biological knowledge.

4. Biology

   Biology-guided Neural Network for Species Classification

   M. Elhamod, M. Maruf, PK Mandke, and 1 more author

   In Integrative And Comparative Biology Workshop, 2021
5. AAAI 21

   

   PID‑GAN: A GAN Framework based on a Physics‑informed Discriminator for Uncertainty Quantification with Physics

   Arka Daw^*, M. Maruf^*, and Anuj Karpatne

   In 2nd symposium on Science‑Guided AI (SGAI) at (AAAI), 2021

2020

1. SDM 20

   

   Maximizing cohesion and separation in graph representation learning: A distance-aware negative sampling approach

   M. Maruf, and Anuj Karpatne

   In Proceedings of the 2021 SIAM International Conference on Data Mining, 2020

   Abs arXiv Code

   The objective of unsupervised graph representation learning (GRL) is to learn a low-dimensional space of node embeddings that reflect the structure of a given unlabeled graph. Existing algorithms for this task rely on negative sampling objectives that maximize the similarity in node embeddings at nearby nodes (referred to as "cohesion") by maintaining positive and negative corpus of node pairs. While positive samples are drawn from node pairs that co-occur in short random walks, conventional approaches construct negative corpus by uniformly sampling random pairs, thus ignoring valuable information about structural dissimilarity among distant node pairs (referred to as "separation"). In this paper, we present a novel Distance-aware Negative Sampling (DNS) which maximizes the separation of distant node-pairs while maximizing cohesion at nearby node-pairs by setting the negative sampling probability proportional to the pair-wise shortest distances. Our approach can be used in conjunction with any GRL algorithm and we demonstrate the efficacy of our approach over baseline negative sampling methods over downstream node classification tasks on a number of benchmark datasets and GRL algorithms.

2. ML4PS NeurIPS

   Physics-Informed Discriminator (PID) for Conditional Generative Adversarial Nets

   Arka Daw, M. Maruf, and Anuj Karpatne

   In Machine Learning and Physical Sciences Workshop at Neural Information Processing Systems (NeurIPS), 2020
3. PIML 20

   Informing Neural Networks for Drug Effect Prediction Using Biological Knowledge of Protein-Protein Interactions

   M. Maruf, and Anuj Karpatne

   In 3rd Physics Informed Machine Learning (PIML) Workshop, 2020

2019

1. Genomics 19

   iRSpot-SF: Prediction of recombination hotspots by incorporating sequence-based features into Pseudo components

   M. Maruf, and Swakkhar Shatabda

   Genomics, 2019

   Website