Earth Microbiome Project Case Study - Notebook#
This notebook is an end-to-end demo of the Vuegen Python package, starting from content generation. This sofwtare automates the creation of reports based on a directory with plots, dataframes, and other files in different formats. A YAML configuration file is generated from the directory to define the structure of the report. Users can customize the report by modifying the configuration file, or they can create their own configuration file instead of passing a directory as input.
The configuration file specifies the structure of the report, including sections, subsections, and various components such as plots, dataframes, markdown, html, and API calls. Reports can be generated in various formats, including documents (PDF, HTML, DOCX, ODT), presentations (PPTX, Reveal.js), notebooks (Jupyter) or Streamlit web applications.
An overview of the VueGen workflow is shown in the figure below:
About the data#
The Earth Microbiome Project (EMP) is a systematic attempt to characterize global microbial taxonomic and functional diversity for the benefit of the planet and humankind. It aimed to sample the Earth’s microbial communities at an unprecedented scale in order to advance our understanding of the organizing biogeographic principles that govern microbial community structure. The EMP dataset is generated from samples that individual researchers have compiled and contributed to the EMP. The result is both a reference database giving global context to DNA sequence data and a framework for incorporating data from future studies, fostering increasingly complete characterization of Earth’s microbial diversity.
The data and scripts for this project are available on the EMP GitHub repository, and addtional details can be found in the original article by Thompson et al. (2017).
Notebook structure#
First, we will set up the work environment by installing the necessary packages and importing the required libraries. Then, we will create various plots and tables on the exploratory data analysis, metagenomics, and network analysis sections. Next, we will create a report using Vuegen, including the plots and tables generated in the previous sections. Finally, we will extend the report by modifying the configuration file to include additional components.
Credits and Contributors#
This notebook was created by Sebastián Ayala-Ruano under the supervision of Henry Webel and Alberto Santos, head of the Multiomics Network Analytics Group (MoNA) at the Novo Nordisk Foundation Center for Biosustainability (DTU Biosustain).
Part of this notebook’s code was inspired by the scripts and notebooks from the EMP GitHub repository.
You can find more details about the project in this GitHub repository.
0. Work environment setup and helper functions#
0.1. Installing libraries and creating global variables for platform and working directory#
To run this notebook locally, you should create a virtual environment with the required libraries. If you are running this notebook on Google Colab, everything should be set.
# Vuegen library
%pip install vuegen
Show code cell output
Hide code cell output
Requirement already satisfied: vuegen in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (0.6.0)
Requirement already satisfied: altair[save] in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (6.0.0)
Requirement already satisfied: dataframe-image<0.3.0,>=0.2.6 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (0.2.7)
Requirement already satisfied: itables<3.0.0,>=2.2.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (2.7.0)
Requirement already satisfied: kaleido<0.3.0,>=0.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (0.2.1)
Requirement already satisfied: matplotlib<4.0.0,>=3.9.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (3.10.8)
Requirement already satisfied: nbclient<0.11.0,>=0.10.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (0.10.4)
Requirement already satisfied: nbformat<6.0.0,>=5.10.4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (5.10.4)
Requirement already satisfied: openpyxl<4.0.0,>=3.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (3.1.5)
Requirement already satisfied: pandas<3.0.0,>=2.2.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pandas[parquet]<3.0.0,>=2.2.2->vuegen) (2.3.3)
Requirement already satisfied: plotly<6.0.0,>=5.15.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (5.24.1)
Requirement already satisfied: pyvis<0.4.0,>=0.3.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (0.3.2)
Requirement already satisfied: pyyaml<7.0.0,>=6.0.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (6.0.3)
Requirement already satisfied: quarto-cli in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (1.8.27)
Requirement already satisfied: streamlit<2.0.0,>=1.39.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (1.54.0)
Requirement already satisfied: streamlit-aggrid in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (1.2.1.post2)
Requirement already satisfied: vl-convert-python<2.0.0,>=1.7.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (1.9.0.post1)
Requirement already satisfied: xlrd<3.0.0,>=2.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from vuegen) (2.0.2)
Requirement already satisfied: nbconvert>=5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (7.17.0)
Requirement already satisfied: aiohttp>=3.10.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (3.13.3)
Requirement already satisfied: requests in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (2.32.5)
Requirement already satisfied: pillow in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (12.1.0)
Requirement already satisfied: packaging in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (26.0)
Requirement already satisfied: mistune in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (3.2.0)
Requirement already satisfied: lxml in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (6.0.2)
Requirement already satisfied: beautifulsoup4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (4.14.3)
Requirement already satisfied: cssutils in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (2.11.1)
Requirement already satisfied: playwright in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (1.58.0)
Requirement already satisfied: cssselect in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from dataframe-image<0.3.0,>=0.2.6->vuegen) (1.4.0)
Requirement already satisfied: contourpy>=1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.2->vuegen) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.2->vuegen) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.2->vuegen) (4.61.1)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.2->vuegen) (1.4.9)
Requirement already satisfied: numpy>=1.23 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.2->vuegen) (2.4.2)
Requirement already satisfied: pyparsing>=3 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.2->vuegen) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib<4.0.0,>=3.9.2->vuegen) (2.9.0.post0)
Requirement already satisfied: jupyter-client>=6.1.12 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbclient<0.11.0,>=0.10.0->vuegen) (8.8.0)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbclient<0.11.0,>=0.10.0->vuegen) (5.9.1)
Requirement already satisfied: traitlets>=5.4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbclient<0.11.0,>=0.10.0->vuegen) (5.14.3)
Requirement already satisfied: fastjsonschema>=2.15 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbformat<6.0.0,>=5.10.4->vuegen) (2.21.2)
Requirement already satisfied: jsonschema>=2.6 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbformat<6.0.0,>=5.10.4->vuegen) (4.26.0)
Requirement already satisfied: et-xmlfile in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from openpyxl<4.0.0,>=3.1.5->vuegen) (2.0.0)
Requirement already satisfied: pytz>=2020.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pandas<3.0.0,>=2.2.2->pandas[parquet]<3.0.0,>=2.2.2->vuegen) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pandas<3.0.0,>=2.2.2->pandas[parquet]<3.0.0,>=2.2.2->vuegen) (2025.3)
Requirement already satisfied: pyarrow>=10.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pandas[parquet]<3.0.0,>=2.2.2->vuegen) (23.0.0)
Requirement already satisfied: tenacity>=6.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from plotly<6.0.0,>=5.15.0->vuegen) (9.1.4)
Requirement already satisfied: ipython>=5.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pyvis<0.4.0,>=0.3.2->vuegen) (9.10.0)
Requirement already satisfied: jinja2>=2.9.6 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pyvis<0.4.0,>=0.3.2->vuegen) (3.1.6)
Requirement already satisfied: jsonpickle>=1.4.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pyvis<0.4.0,>=0.3.2->vuegen) (4.1.1)
Requirement already satisfied: networkx>=1.11 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pyvis<0.4.0,>=0.3.2->vuegen) (3.6.1)
Requirement already satisfied: blinker<2,>=1.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (1.9.0)
Requirement already satisfied: cachetools<7,>=5.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (6.2.6)
Requirement already satisfied: click<9,>=7.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (8.3.1)
Requirement already satisfied: gitpython!=3.1.19,<4,>=3.0.7 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (3.1.46)
Requirement already satisfied: pydeck<1,>=0.8.0b4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (0.9.1)
Requirement already satisfied: protobuf<7,>=3.20 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (6.33.5)
Requirement already satisfied: toml<2,>=0.10.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (0.10.2)
Requirement already satisfied: tornado!=6.5.0,<7,>=6.0.3 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (6.5.4)
Requirement already satisfied: typing-extensions<5,>=4.10.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (4.15.0)
Requirement already satisfied: watchdog<7,>=2.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit<2.0.0,>=1.39.0->vuegen) (6.0.0)
Requirement already satisfied: narwhals>=1.27.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from altair[save]->vuegen) (2.16.0)
Requirement already satisfied: gitdb<5,>=4.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from gitpython!=3.1.19,<4,>=3.0.7->streamlit<2.0.0,>=1.39.0->vuegen) (4.0.12)
Requirement already satisfied: smmap<6,>=3.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from gitdb<5,>=4.0.1->gitpython!=3.1.19,<4,>=3.0.7->streamlit<2.0.0,>=1.39.0->vuegen) (5.0.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from requests->dataframe-image<0.3.0,>=0.2.6->vuegen) (3.4.4)
Requirement already satisfied: idna<4,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from requests->dataframe-image<0.3.0,>=0.2.6->vuegen) (3.11)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from requests->dataframe-image<0.3.0,>=0.2.6->vuegen) (2.6.3)
Requirement already satisfied: certifi>=2017.4.17 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from requests->dataframe-image<0.3.0,>=0.2.6->vuegen) (2026.1.4)
Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from aiohttp>=3.10.2->dataframe-image<0.3.0,>=0.2.6->vuegen) (2.6.1)
Requirement already satisfied: aiosignal>=1.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from aiohttp>=3.10.2->dataframe-image<0.3.0,>=0.2.6->vuegen) (1.4.0)
Requirement already satisfied: attrs>=17.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from aiohttp>=3.10.2->dataframe-image<0.3.0,>=0.2.6->vuegen) (25.4.0)
Requirement already satisfied: frozenlist>=1.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from aiohttp>=3.10.2->dataframe-image<0.3.0,>=0.2.6->vuegen) (1.8.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from aiohttp>=3.10.2->dataframe-image<0.3.0,>=0.2.6->vuegen) (6.7.1)
Requirement already satisfied: propcache>=0.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from aiohttp>=3.10.2->dataframe-image<0.3.0,>=0.2.6->vuegen) (0.4.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from aiohttp>=3.10.2->dataframe-image<0.3.0,>=0.2.6->vuegen) (1.22.0)
Requirement already satisfied: decorator>=4.3.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (5.2.1)
Requirement already satisfied: ipython-pygments-lexers>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (1.1.1)
Requirement already satisfied: jedi>=0.18.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (0.19.2)
Requirement already satisfied: matplotlib-inline>=0.1.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (0.2.1)
Requirement already satisfied: pexpect>4.3 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (4.9.0)
Requirement already satisfied: prompt_toolkit<3.1.0,>=3.0.41 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (3.0.52)
Requirement already satisfied: pygments>=2.11.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (2.19.2)
Requirement already satisfied: stack_data>=0.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (0.6.3)
Requirement already satisfied: wcwidth in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from prompt_toolkit<3.1.0,>=3.0.41->ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (0.6.0)
Requirement already satisfied: parso<0.9.0,>=0.8.4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jedi>=0.18.1->ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (0.8.6)
Requirement already satisfied: MarkupSafe>=2.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jinja2>=2.9.6->pyvis<0.4.0,>=0.3.2->vuegen) (3.0.3)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat<6.0.0,>=5.10.4->vuegen) (2025.9.1)
Requirement already satisfied: referencing>=0.28.4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat<6.0.0,>=5.10.4->vuegen) (0.37.0)
Requirement already satisfied: rpds-py>=0.25.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema>=2.6->nbformat<6.0.0,>=5.10.4->vuegen) (0.30.0)
Requirement already satisfied: pyzmq>=25.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-client>=6.1.12->nbclient<0.11.0,>=0.10.0->vuegen) (27.1.0)
Requirement already satisfied: platformdirs>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-core!=5.0.*,>=4.12->nbclient<0.11.0,>=0.10.0->vuegen) (4.5.1)
Requirement already satisfied: bleach!=5.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from bleach[css]!=5.0.0->nbconvert>=5->dataframe-image<0.3.0,>=0.2.6->vuegen) (6.3.0)
Requirement already satisfied: defusedxml in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbconvert>=5->dataframe-image<0.3.0,>=0.2.6->vuegen) (0.7.1)
Requirement already satisfied: jupyterlab-pygments in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbconvert>=5->dataframe-image<0.3.0,>=0.2.6->vuegen) (0.3.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from nbconvert>=5->dataframe-image<0.3.0,>=0.2.6->vuegen) (1.5.1)
Requirement already satisfied: webencodings in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from bleach!=5.0.0->bleach[css]!=5.0.0->nbconvert>=5->dataframe-image<0.3.0,>=0.2.6->vuegen) (0.5.1)
Requirement already satisfied: tinycss2<1.5,>=1.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from bleach[css]!=5.0.0->nbconvert>=5->dataframe-image<0.3.0,>=0.2.6->vuegen) (1.4.0)
Requirement already satisfied: ptyprocess>=0.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pexpect>4.3->ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (0.7.0)
Requirement already satisfied: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib<4.0.0,>=3.9.2->vuegen) (1.17.0)
Requirement already satisfied: executing>=1.2.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (2.2.1)
Requirement already satisfied: asttokens>=2.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (3.0.1)
Requirement already satisfied: pure-eval in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from stack_data>=0.6.0->ipython>=5.3.0->pyvis<0.4.0,>=0.3.2->vuegen) (0.2.3)
Requirement already satisfied: soupsieve>=1.6.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from beautifulsoup4->dataframe-image<0.3.0,>=0.2.6->vuegen) (2.8.3)
Requirement already satisfied: more-itertools in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from cssutils->dataframe-image<0.3.0,>=0.2.6->vuegen) (10.8.0)
Requirement already satisfied: pyee<14,>=13 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from playwright->dataframe-image<0.3.0,>=0.2.6->vuegen) (13.0.0)
Requirement already satisfied: greenlet<4.0.0,>=3.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from playwright->dataframe-image<0.3.0,>=0.2.6->vuegen) (3.3.1)
Requirement already satisfied: jupyter in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from quarto-cli->vuegen) (1.1.1)
Requirement already satisfied: wheel in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from quarto-cli->vuegen) (0.40.0)
Requirement already satisfied: notebook in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter->quarto-cli->vuegen) (7.5.3)
Requirement already satisfied: jupyter-console in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter->quarto-cli->vuegen) (6.6.3)
Requirement already satisfied: ipykernel in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter->quarto-cli->vuegen) (7.2.0)
Requirement already satisfied: ipywidgets in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter->quarto-cli->vuegen) (8.1.8)
Requirement already satisfied: jupyterlab in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter->quarto-cli->vuegen) (4.5.3)
Requirement already satisfied: comm>=0.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipykernel->jupyter->quarto-cli->vuegen) (0.2.3)
Requirement already satisfied: debugpy>=1.6.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipykernel->jupyter->quarto-cli->vuegen) (1.8.20)
Requirement already satisfied: nest-asyncio>=1.4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipykernel->jupyter->quarto-cli->vuegen) (1.6.0)
Requirement already satisfied: psutil>=5.7 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipykernel->jupyter->quarto-cli->vuegen) (7.2.2)
Requirement already satisfied: widgetsnbextension~=4.0.14 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipywidgets->jupyter->quarto-cli->vuegen) (4.0.15)
Requirement already satisfied: jupyterlab_widgets~=3.0.15 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from ipywidgets->jupyter->quarto-cli->vuegen) (3.0.16)
Requirement already satisfied: async-lru>=1.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab->jupyter->quarto-cli->vuegen) (2.1.0)
Requirement already satisfied: httpx<1,>=0.25.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab->jupyter->quarto-cli->vuegen) (0.28.1)
Requirement already satisfied: jupyter-lsp>=2.0.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab->jupyter->quarto-cli->vuegen) (2.3.0)
Requirement already satisfied: jupyter-server<3,>=2.4.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab->jupyter->quarto-cli->vuegen) (2.17.0)
Requirement already satisfied: jupyterlab-server<3,>=2.28.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab->jupyter->quarto-cli->vuegen) (2.28.0)
Requirement already satisfied: notebook-shim>=0.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab->jupyter->quarto-cli->vuegen) (0.2.4)
Requirement already satisfied: setuptools>=41.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab->jupyter->quarto-cli->vuegen) (82.0.0)
Requirement already satisfied: anyio in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from httpx<1,>=0.25.0->jupyterlab->jupyter->quarto-cli->vuegen) (4.12.1)
Requirement already satisfied: httpcore==1.* in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from httpx<1,>=0.25.0->jupyterlab->jupyter->quarto-cli->vuegen) (1.0.9)
Requirement already satisfied: h11>=0.16 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.25.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.16.0)
Requirement already satisfied: argon2-cffi>=21.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (25.1.0)
Requirement already satisfied: jupyter-events>=0.11.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.12.0)
Requirement already satisfied: jupyter-server-terminals>=0.4.4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.5.4)
Requirement already satisfied: overrides>=5.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (7.7.0)
Requirement already satisfied: prometheus-client>=0.9 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.24.1)
Requirement already satisfied: send2trash>=1.8.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (2.1.0)
Requirement already satisfied: terminado>=0.8.3 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.18.1)
Requirement already satisfied: websocket-client>=1.7 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (1.9.0)
Requirement already satisfied: babel>=2.10 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab-server<3,>=2.28.0->jupyterlab->jupyter->quarto-cli->vuegen) (2.18.0)
Requirement already satisfied: json5>=0.9.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyterlab-server<3,>=2.28.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.13.0)
Requirement already satisfied: argon2-cffi-bindings in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (25.1.0)
Requirement already satisfied: python-json-logger>=2.0.4 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (4.0.0)
Requirement already satisfied: rfc3339-validator in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.1.4)
Requirement already satisfied: rfc3986-validator>=0.1.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (0.1.1)
Requirement already satisfied: fqdn in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (1.5.1)
Requirement already satisfied: isoduration in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (20.11.0)
Requirement already satisfied: jsonpointer>1.13 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (3.0.0)
Requirement already satisfied: rfc3987-syntax>=1.1.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (1.1.0)
Requirement already satisfied: uri-template in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (1.3.0)
Requirement already satisfied: webcolors>=24.6.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (25.10.0)
Requirement already satisfied: lark>=1.2.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from rfc3987-syntax>=1.1.0->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (1.3.1)
Requirement already satisfied: cffi>=1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (2.0.0)
Requirement already satisfied: pycparser in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (3.0)
Requirement already satisfied: arrow>=0.15.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.11.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter->quarto-cli->vuegen) (1.4.0)
Requirement already satisfied: python-decouple in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from streamlit-aggrid->vuegen) (3.8)
Note: you may need to restart the kernel to use updated packages.
# Libraries for the notebook
%pip install cartopy seaborn biom-format
Show code cell output
Hide code cell output
Collecting cartopy
Downloading cartopy-0.25.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (6.1 kB)
Collecting seaborn
Downloading seaborn-0.13.2-py3-none-any.whl.metadata (5.4 kB)
Collecting biom-format
Downloading biom_format-2.1.17-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Requirement already satisfied: numpy>=1.23 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from cartopy) (2.4.2)
Requirement already satisfied: matplotlib>=3.6 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from cartopy) (3.10.8)
Collecting shapely>=2.0 (from cartopy)
Downloading shapely-2.1.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (6.8 kB)
Requirement already satisfied: packaging>=21 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from cartopy) (26.0)
Collecting pyshp>=2.3 (from cartopy)
Downloading pyshp-3.0.3-py3-none-any.whl.metadata (65 kB)
Collecting pyproj>=3.3.1 (from cartopy)
Downloading pyproj-3.7.2-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (31 kB)
Requirement already satisfied: pandas>=1.2 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from seaborn) (2.3.3)
Requirement already satisfied: click in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from biom-format) (8.3.1)
Collecting scipy>=1.3.1 (from biom-format)
Downloading scipy-1.17.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (62 kB)
Collecting h5py (from biom-format)
Downloading h5py-3.15.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Requirement already satisfied: contourpy>=1.0.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib>=3.6->cartopy) (1.3.3)
Requirement already satisfied: cycler>=0.10 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib>=3.6->cartopy) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib>=3.6->cartopy) (4.61.1)
Requirement already satisfied: kiwisolver>=1.3.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib>=3.6->cartopy) (1.4.9)
Requirement already satisfied: pillow>=8 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib>=3.6->cartopy) (12.1.0)
Requirement already satisfied: pyparsing>=3 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib>=3.6->cartopy) (3.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from matplotlib>=3.6->cartopy) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pandas>=1.2->seaborn) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pandas>=1.2->seaborn) (2025.3)
Requirement already satisfied: certifi in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from pyproj>=3.3.1->cartopy) (2026.1.4)
Requirement already satisfied: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib>=3.6->cartopy) (1.17.0)
Downloading cartopy-0.25.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (11.8 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/11.8 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 120.4 MB/s 0:00:00
?25hDownloading seaborn-0.13.2-py3-none-any.whl (294 kB)
Downloading biom_format-2.1.17-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/12.3 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.3/12.3 MB 164.1 MB/s 0:00:00
?25hDownloading pyproj-3.7.2-cp311-cp311-manylinux_2_28_x86_64.whl (9.5 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/9.5 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.5/9.5 MB 149.8 MB/s 0:00:00
?25h
Downloading pyshp-3.0.3-py3-none-any.whl (58 kB)
Downloading scipy-1.17.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (35.1 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/35.1 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 35.1/35.1 MB 165.6 MB/s 0:00:00
?25h
Downloading shapely-2.1.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 152.2 MB/s 0:00:00
?25hDownloading h5py-3.15.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (4.7 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/4.7 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 146.8 MB/s 0:00:00
?25h
Installing collected packages: shapely, scipy, pyshp, pyproj, h5py, seaborn, cartopy, biom-format
?25l
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/8 [shapely]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/8 [scipy]
━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━ 3/8 [pyproj]
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 4/8 [h5py]
━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━ 5/8 [seaborn]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 6/8 [cartopy]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 6/8 [cartopy]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 7/8 [biom-format]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━ 7/8 [biom-format]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8/8 [biom-format]
Successfully installed biom-format-2.1.17 cartopy-0.25.0 h5py-3.15.1 pyproj-3.7.2 pyshp-3.0.3 scipy-1.17.0 seaborn-0.13.2 shapely-2.1.2
Note: you may need to restart the kernel to use updated packages.
Show code cell content
Hide code cell content
import os
IN_COLAB = "COLAB_GPU" in os.environ
Show code cell content
Hide code cell content
# Optional library to launch a streamlit app from colab
if IN_COLAB:
!npm install localtunnel
Show code cell content
Hide code cell content
# Output directory
base_output_dir = "./example_data/Earth_microbiome_vuegen_demo_notebook/"
0.2. Importing libraries#
Show code cell content
Hide code cell content
# Imports
import os
import yaml
import requests
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import plotly.graph_objects as go
import plotly.express as px
import altair as alt
import seaborn as sns
import biom
import networkx as nx
from vuegen import report_generator
from vuegen.utils import get_logger, load_yaml_config
if IN_COLAB:
import urllib
0.3. Creating working directory#
Show code cell content
Hide code cell content
os.makedirs(base_output_dir, exist_ok=True)
0.4. Markdown project description#
Create a markdown file with the project description to be displayed in the main page of the report. It is also possible to do this for the sections and subsections of the report by creating a description.md file in the corresponding directory.
Show code cell content
Hide code cell content
# Create project description
project_description = """
The Earth Microbiome Project (EMP) is a systematic attempt to characterize global microbial taxonomic and functional diversity for the benefit of the planet and humankind.
It aimed to sample the Earth’s microbial communities at an unprecedented scale in order to advance our understanding of the organizing biogeographic principles that govern microbial community structure.
The EMP dataset is generated from samples that individual researchers have compiled and contributed to the EMP.
The result is both a reference database giving global context to DNA sequence data and a framework for incorporating data from future studies, fostering increasingly complete characterization of Earth’s microbial diversity.
You can find more information about the Earth Microbiome Project at https://earthmicrobiome.org/ and in the [original article](https://www.nature.com/articles/nature24621).
"""
# Write the description to a file
with open(os.path.join(base_output_dir, "description.md"), "w") as file:
file.write(project_description)
0.5. Helper functions#
Show code cell content
Hide code cell content
def get_empo_cat_color(empocat=None, returndict=False):
"""returns standard empo colors for EMP figures
input:
empocat- empo category name (string)
returndict- the user needs the dictionary mapping category to color (boolean)
output: either a color for passed empocat or the dictionay if returndict=True"""
# hex codes for matplotlib colors are described here:
# https://github.com/matplotlib/matplotlib/blob/cf83cd5642506ef808853648b9eb409f8dbd6ff3/lib/matplotlib/_color_data.py
empo_cat_color = {
"EMP sample": "#929591", # 'grey'
"Host-associated": "#fb9a99",
"Free-living": "#e31a1c",
"Animal": "#b2df8a",
"Plant": "#33a02c",
"Non-saline": "#a6cee3",
"Saline": "#1f78b4",
"Aerosol (non-saline)": "#d3d3d3", # 'lightgrey'
"Animal corpus": "#ffff00", # 'yellow'
"Animal distal gut": "#8b4513", # 'saddlebrown'
"Animal proximal gut": "#d2b48c", # 'tan'
"Animal secretion": "#f4a460", # 'sandybrown'
"Animal surface": "#b8860b", # 'darkgoldenrod'
"Hypersaline (saline)": "#87cefa", # 'lightskyblue'
"Intertidal (saline)": "#afeeee", # 'paleturquoise'
"Mock community": "#ff00ff", # 'fuchsia'
"Plant corpus": "#7cfc00", # 'lawngreen'
"Plant rhizosphere": "#006400", # 'darkgreen'
"Plant surface": "#00fa9a", # 'mediumspringgreen'
"Sediment (non-saline)": "#ffa07a", # 'lightsalmon'
"Sediment (saline)": "#ff6347", # 'tomato'
"Soil (non-saline)": "#ff0000", # 'red'
"Sterile water blank": "#ee82ee", # 'violet'
"Surface (non-saline)": "#000000", # 'black'
"Surface (saline)": "#696969", # 'dimgrey'
"Water (non-saline)": "#000080", # 'navy'
"Water (saline)": "#4169e1", # 'royalblue'
}
if returndict == True:
return empo_cat_color
else:
return empo_cat_color[empocat]
1. Exploratory data analysis#
1.1. Global scope of sample provenance#
Samples come from 7 continents, 43 countries, 21 biomes (ENVO), 92 environmental features (ENVO), and 17 environments (EMPO).
1.1.1. Load data and create section and subsection directories#
Show code cell content
Hide code cell content
# Create the output directory for the EDA section and sample provenance subsection
sample_prov_output_dir = os.path.join(
base_output_dir, "1_Exploratory_data_analysis/1_sample_exploration/"
)
os.makedirs(sample_prov_output_dir, exist_ok=True)
# Load data and filter out control samples
metadata_mapping = "https://raw.githubusercontent.com//biocore/emp/master/data/mapping-files/emp_qiime_mapping_release1.tsv"
metadata_mapping_df = pd.read_table(metadata_mapping, index_col=0)
metadata_mapping_df = metadata_mapping_df[metadata_mapping_df["empo_1"] != "Control"]
1.1.2. Random subset of the metadata dataframe#
Show code cell content
Hide code cell content
# Obtain a random sample of the metadata df with a random seed
sample_metadata_mapping_df = metadata_mapping_df.sample(100, random_state=42)
# Export the sample df as a CSV file
sample_metadata_mapping_df.to_csv(
f"{sample_prov_output_dir}/1_metadata_random_subset.csv"
)
1.1.3. Static map with matplotlib#
Show code cell source
Hide code cell source
# Extract Animal dataset
animal_df = metadata_mapping_df[metadata_mapping_df["empo_2"] == "Animal"]
# Unique subcategories in empo_3
animal_empo3 = animal_df["empo_3"].unique()
# Create a figure with Cartopy map projection
fig, ax = plt.subplots(
figsize=(12, 8), dpi=300, subplot_kw={"projection": ccrs.PlateCarree()}
)
# Add features to the map
ax.add_feature(cfeature.BORDERS, edgecolor="white", linewidth=0.5)
ax.add_feature(cfeature.LAND, edgecolor="white", facecolor="lightgray", linewidth=0.5)
ax.add_feature(cfeature.COASTLINE, edgecolor="white", linewidth=0.5)
# Set extent (global map)
ax.set_extent([-180, 180, -90, 90])
# Plot each subcategory with a unique color (unfilled circles with borders)
for empo3 in animal_empo3:
subset = animal_df[animal_df["empo_3"] == empo3]
color = get_empo_cat_color(empo3) # Get color for category
ax.scatter(
subset["longitude_deg"],
subset["latitude_deg"],
color="none",
edgecolors=color,
linewidth=1.5,
label=empo3,
s=40,
transform=ccrs.PlateCarree(),
zorder=2,
)
# Add legend with updated labels
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles, labels, loc="lower center", ncol=2, fontsize=10)
# Save the figure
animal_map_out_path = os.path.join(sample_prov_output_dir, "2_animal_samples_map.png")
plt.savefig(animal_map_out_path, dpi=300, bbox_inches="tight")
/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages/cartopy/io/__init__.py:242: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/110m_physical/ne_110m_land.zip
warnings.warn(f'Downloading: {url}', DownloadWarning)
/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages/cartopy/io/__init__.py:242: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/110m_cultural/ne_110m_admin_0_boundary_lines_land.zip
warnings.warn(f'Downloading: {url}', DownloadWarning)
/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages/cartopy/io/__init__.py:242: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/110m_physical/ne_110m_coastline.zip
warnings.warn(f'Downloading: {url}', DownloadWarning)
1.1.4. Interactive map with plotly#
Show code cell source
Hide code cell source
# Extract Plant dataset
plant_df = metadata_mapping_df[metadata_mapping_df["empo_2"] == "Plant"]
# Unique subcategories in empo_3
plant_empo3 = plant_df["empo_3"].unique()
# Create a base map figure
fig = go.Figure()
# Plot each subcategory with unique color (unfilled circles with borders)
for empo3 in plant_empo3:
subset = plant_df[plant_df["empo_3"] == empo3]
color = get_empo_cat_color(empo3) # Get color for category
fig.add_trace(
go.Scattergeo(
lon=subset["longitude_deg"],
lat=subset["latitude_deg"],
mode="markers",
marker=dict(
symbol="circle-open", # Unfilled circle
color=color,
size=6, # Marker size
line=dict(
width=1.5, color=color
), # Border color matches category color
),
name=empo3,
)
)
# Update map layout (fixes horizontal blank space)
fig.update_layout(
geo=dict(
landcolor="rgb(217, 217, 217)", # Light gray land color
showcountries=True, # Enable country borders
countrycolor="white", # Set country borders to white
countrywidth=0.5, # Border thickness
coastlinecolor="white",
fitbounds="locations", # Focuses only on data points
lataxis=dict(range=[-60, 85], showgrid=False), # Custom latitude range
lonaxis=dict(range=[-180, 180], showgrid=False), # Custom longitude range
),
autosize=False,
width=800, # Adjust width to remove blank space
height=600, # Keep a balanced height
margin=dict(l=10, r=10, t=10, b=10), # Reduce margins to keep map centered
legend=dict(
title="",
font=dict(size=16), # Legend font size
x=0.5, # Center legend horizontally
xanchor="center",
yanchor="top",
orientation="h", # Horizontal legend layout
),
)
# Save the figure as SVG and JSON
# plant_map_out_path = os.path.join(sample_prov_output_dir, "2_plant_samples_map.svg")
# fig.write_image(plant_map_out_path, scale=2)
plant_map_json_path = os.path.join(sample_prov_output_dir, "3_plant_samples_map.json")
fig.write_json(plant_map_json_path)
# Show the figure
fig.show(renderer="svg")
1.1.5. Interactive map with Altair#
Show code cell source
Hide code cell source
# Extract Saline dataset
saline_df = metadata_mapping_df[metadata_mapping_df["empo_2"] == "Saline"].copy()
# Create a dictionary for simplified category names for the legend
simplified_category_names = {
"Water (saline)": "Water",
"Sediment (saline)": "Sediment",
"Surface (saline)": "Surface",
"Hypersaline (saline)": "Hypersaline",
}
# Simplify the empo_3 names in the DataFrame for legend
saline_df["simplified_empo_3"] = saline_df["empo_3"].apply(
lambda x: simplified_category_names.get(x, x)
)
# Apply the get_empo_cat_color function to generate the color column
saline_df["color"] = saline_df["empo_3"].apply(get_empo_cat_color)
# Create the base world map (use the CDN URL or a different base map if you prefer)
countries = alt.topo_feature(
"https://cdn.jsdelivr.net/npm/world-atlas@2/countries-110m.json", "countries"
)
# Create a background map of countries
background_map = (
alt.Chart(countries)
.mark_geoshape(fill="lightgray", stroke="white")
.project("equirectangular")
.properties(width=800, height=400)
)
# Create the points for saline samples with custom colors
saline_points = (
alt.Chart(saline_df)
.mark_point(
size=50,
shape="circle",
filled=False,
)
.encode(
longitude="longitude_deg:Q",
latitude="latitude_deg:Q",
color=alt.Color(
"simplified_empo_3:N",
scale=alt.Scale(
domain=list(saline_df["simplified_empo_3"].unique()),
range=[get_empo_cat_color(cat) for cat in saline_df["empo_3"].unique()],
),
legend=alt.Legend(
title="", orient="bottom", symbolSize=120, labelFontSize=16
),
),
tooltip=["latitude_deg", "longitude_deg", "empo_3"],
)
)
# Overlay the points on the world map
map_with_points = background_map + saline_points
# Save as JSON and
saline_json_path = os.path.join(sample_prov_output_dir, "4_saline_samples_map.json")
with open(saline_json_path, "w") as f:
f.write(map_with_points.to_json())
# saline_png_path = os.path.join(sample_prov_output_dir, "5_saline_samples_map.png")
# map_with_points.save(saline_png_path)
# Display the map
map_with_points
2. Metagenomics#
2.1. Alpha diversity#
The alpha diversity was measured as number of observed 90bp tag sequences (richness), in n = 23,828 biologically independent samples as a function of environment. Tag sequence counts were subsampled to 5,000 observations.
2.1.1. Load data and create section and subsection directories#
Show code cell content
Hide code cell content
# Create the output directory for the metagenomics section and alpha diversity subsection
alpha_div_output_dir = os.path.join(base_output_dir, "2_Metagenomics/1_alpha_diversity")
os.makedirs(alpha_div_output_dir, exist_ok=True)
# Load data
mapping_qc_filt = "https://raw.githubusercontent.com//biocore/emp/master/data/mapping-files/emp_qiime_mapping_qc_filtered.tsv"
mapping_qc_filt_df = pd.read_csv(
mapping_qc_filt, sep="\t", index_col=0, header=0
).sort_index()
Show code cell content
Hide code cell content
# Define colors of host associated and free living categories
colorsHA = {
"Animal corpus": get_empo_cat_color("Animal corpus"),
"Plant corpus": get_empo_cat_color("Plant corpus"),
"Animal secretion": get_empo_cat_color("Animal secretion"),
"Plant surface": get_empo_cat_color("Plant surface"),
"Animal proximal gut": get_empo_cat_color("Animal proximal gut"),
"Animal surface": get_empo_cat_color("Animal surface"),
"Animal distal gut": get_empo_cat_color("Animal distal gut"),
"Plant rhizosphere": get_empo_cat_color("Plant rhizosphere"),
}
colorsFL = {
"Water (saline)": get_empo_cat_color("Water (saline)"),
"Aerosol (non-saline)": get_empo_cat_color("Aerosol (non-saline)"),
"Hypersaline (saline)": get_empo_cat_color("Hypersaline (saline)"),
"Surface (non-saline)": get_empo_cat_color("Surface (non-saline)"),
"Surface (saline)": get_empo_cat_color("Surface (saline)"),
"Water (non-saline)": get_empo_cat_color("Water (non-saline)"),
"Sediment (saline)": get_empo_cat_color("Sediment (saline)"),
"Soil (non-saline)": get_empo_cat_color("Soil (non-saline)"),
"Sediment (non-saline)": get_empo_cat_color("Sediment (non-saline)"),
}
2.1.2. Static box plot with matplotlib#
Show code cell source
Hide code cell source
# Ensure y variable is numeric to avoid aggregation errors
mapping_qc_filt_df["adiv_observed_otus"] = pd.to_numeric(
mapping_qc_filt_df["adiv_observed_otus"], errors="coerce"
)
# Get valid categories (only ones in colorsHA)
valid_categories_HA = set(colorsHA.keys())
# Filter dataset to include only valid categories
filtered_data_HA = mapping_qc_filt_df[
(mapping_qc_filt_df["empo_0"] == "EMP sample")
& (mapping_qc_filt_df["empo_3"].isin(valid_categories_HA))
]
# Compute sorted order (only for valid categories)
sorted_order = (
filtered_data_HA.groupby(["empo_3"])["adiv_observed_otus"]
.mean()
.dropna()
.sort_values()
.index.tolist()
)
# Create palette dictionary using only valid categories
palette_dict = {cat: colorsHA[cat] for cat in sorted_order}
# Create figure for Matplotlib
fig = plt.figure(figsize=(16, 8))
# Plot the boxplot and jitter plot
sns.boxplot(
fliersize=0,
x="empo_3",
y="adiv_observed_otus",
hue="empo_3",
linewidth=1,
data=filtered_data_HA,
order=sorted_order,
palette=palette_dict,
)
sns.stripplot(
jitter=True,
x="empo_3",
y="adiv_observed_otus",
data=filtered_data_HA,
order=sorted_order,
color="black",
size=1,
)
# Customize the plot
plt.xticks(rotation=45, ha="right", fontsize=16)
plt.yticks(fontsize=16)
plt.xlabel("")
plt.ylim(0, 3000)
plt.ylabel("Observed tag sequences", fontsize=16)
# Add median line
median = filtered_data_HA["adiv_observed_otus"].median()
plt.axhline(y=median, xmin=0, xmax=1, color="y")
# Adjust layout and save the figure
plt.tight_layout()
fig.patch.set_alpha(0.0)
# Ensure directory exists before saving
os.makedirs(alpha_div_output_dir, exist_ok=True)
# Save figure
alpha_div_box_plot_host_ass = os.path.join(
alpha_div_output_dir, "1_alpha_diversity_host_associated_samples.png"
)
plt.savefig(alpha_div_box_plot_host_ass, dpi=300, bbox_inches="tight")
2.1.3. Interactive box plot with plotly#
Show code cell source
Hide code cell source
# Ensure y variable is numeric to avoid aggregation errors
mapping_qc_filt_df["adiv_observed_otus"] = pd.to_numeric(
mapping_qc_filt_df["adiv_observed_otus"], errors="coerce"
)
# Get valid free-living categories (only ones in colorsFL)
valid_categories_FL = list(colorsFL.keys())
# Filter dataset to include only valid free-living categories
filtered_data_FL = mapping_qc_filt_df[
(mapping_qc_filt_df["empo_0"] == "EMP sample")
& (mapping_qc_filt_df["empo_3"].isin(valid_categories_FL))
]
# Compute sorted order (only for valid categories)
sorted_order_FL = (
filtered_data_FL.groupby(["empo_3"])["adiv_observed_otus"]
.mean()
.dropna()
.sort_values()
.index.tolist()
)
# Create the Plotly figure using boxplot and stripplot (jittered points)
fig = px.box(
filtered_data_FL,
x="empo_3",
y="adiv_observed_otus",
color="empo_3",
category_orders={"empo_3": sorted_order_FL},
color_discrete_map=colorsFL,
labels={"adiv_observed_otus": "Observed tag sequences"},
points=False,
)
# Add jittered points (strip plot)
fig.add_trace(
px.strip(
filtered_data_FL, x="empo_3", y="adiv_observed_otus", stripmode="overlay"
).data[0]
)
# Modify the dot color and size directly inside the add_trace()
fig.data[-1].update(marker=dict(color="black", size=1, opacity=0.7))
# Add median line
median = filtered_data_FL["adiv_observed_otus"].median()
fig.add_hline(y=median, line=dict(color="yellow"))
# Customize the plot
fig.update_layout(
xaxis_title="",
yaxis_title="Observed tag sequences",
xaxis_tickangle=-45,
plot_bgcolor="rgba(0,0,0,0)",
showlegend=False,
height=600,
font=dict(size=14),
)
# Save figure as JSON
alpha_div_box_plot_free_living_json = os.path.join(
alpha_div_output_dir, "2_alpha_diversity_free_living_samples.json"
)
fig.write_json(alpha_div_box_plot_free_living_json)
# Save figure as PNG
# alpha_div_box_plot_free_living_png = os.path.join(alpha_div_output_dir, "3_alpha_div_box_plot_free_living.png")
# fig.write_image(alpha_div_box_plot_free_living_png, width=800, height=600, scale=2)
# Show the plot
fig.show(renderer="png")
2.2. Gene average copy number 16S rRNA#
The average copy number of the 16S rRNA gene was measured in n = 23,828 biologically independent samples as a function of environment.
2.2.1. Load data and create section and subsection directories#
Show code cell content
Hide code cell content
# Create the output directory for the metagenomics section and average copy number subsection
avg_copy_numb_dir = os.path.join(
base_output_dir, "2_Metagenomics/2_average_copy_number"
)
os.makedirs(avg_copy_numb_dir, exist_ok=True)
# Load data
emp_gg_otus_sampsum = "https://raw.githubusercontent.com//biocore/emp/master/data/predicted-rrna-copy-number/emp_cr_gg_13_8.qc_filtered_filt_summary_samplesum.txt"
emp_gg_otus_sampsum_df = pd.read_csv(
emp_gg_otus_sampsum, sep="\t", index_col=0, header=None
).sort_index()
emp_gg_otus_norm_sampsum = "https://raw.githubusercontent.com//biocore/emp/master/data/predicted-rrna-copy-number/emp_cr_gg_13_8.normalized_qcfilt_summary_samplesum.txt"
emp_gg_otus_norm_sampsum_df = pd.read_csv(
emp_gg_otus_norm_sampsum, sep="\t", index_col=0, header=None
).sort_index()
Show code cell content
Hide code cell content
# Create a new df by copying mapping_qc_filt_df
mapping_qc_filt_merged_df = mapping_qc_filt_df.copy()
# Merge new mapping df with emp_gg_otus_sampsum and emp_gg_otus_norm_sampsum
mapping_qc_filt_merged_df["sampsum"] = emp_gg_otus_sampsum_df[1]
mapping_qc_filt_merged_df["normsampsum"] = emp_gg_otus_norm_sampsum_df[1]
mapping_qc_filt_merged_df["copynumberdepletion"] = np.divide(
emp_gg_otus_norm_sampsum_df[1], emp_gg_otus_sampsum_df[1]
)
mapping_qc_filt_merged_df["averagecopy"] = np.divide(
1, np.divide(emp_gg_otus_norm_sampsum_df[1], emp_gg_otus_sampsum_df[1])
)
2.2.2. Static plot with matplotlib#
Show code cell source
Hide code cell source
plt.figure(figsize=(10, 6))
for i in ["Animal", "Non-saline", "Plant", "Saline"]:
plt.hist(
mapping_qc_filt_merged_df[mapping_qc_filt_merged_df.empo_2 == i][
"averagecopy"
].dropna(),
label=i,
bins=200,
linewidth=0,
color=get_empo_cat_color(i),
alpha=0.8,
)
# Customize axes: remove top and right borders
plt.gca().spines["top"].set_visible(False)
plt.gca().spines["right"].set_visible(False)
# Titles and labels
plt.legend(loc=1, prop={"size": 9}, frameon=False)
plt.xlabel("Predicted average community 16S copy number", fontsize=12)
plt.ylabel("Number of samples", fontsize=12)
plt.xticks(fontsize=10)
plt.yticks(fontsize=10)
plt.xlim([0, 8])
plt.tight_layout()
# Save the figure
avg_copy_numb_empo2 = os.path.join(
avg_copy_numb_dir, "1_average_copy_number_emp_ontology_level2.png"
)
plt.savefig(avg_copy_numb_empo2, dpi=300, bbox_inches="tight")
2.2.3. Interactive plot with plotly#
Show code cell source
Hide code cell source
# Create a list to store histogram traces
hist_traces = []
# Loop through unique categories in 'empo_3'
for i in mapping_qc_filt_merged_df.empo_3.dropna().unique():
hist_traces.append(
go.Histogram(
x=mapping_qc_filt_merged_df[mapping_qc_filt_merged_df.empo_3 == i][
"averagecopy"
].dropna(),
name=i, # Legend name
marker=dict(color=get_empo_cat_color(i)), # Assign color
opacity=0.5,
nbinsx=200, # Number of bins
)
)
# Create the figure
fig = go.Figure(hist_traces)
# Update layout to remove top and right borders
fig.update_layout(
xaxis_title="Predicted average community 16S copy number",
yaxis_title="Number of samples",
xaxis=dict(
range=[0, 8],
tickfont=dict(size=10),
showline=True,
linewidth=1,
linecolor="black",
mirror=False,
showgrid=False,
zeroline=False,
),
yaxis=dict(
tickfont=dict(size=10),
showline=True,
linewidth=1,
linecolor="black",
mirror=False,
showgrid=False,
zeroline=False,
),
barmode="overlay",
showlegend=True,
legend=dict(font=dict(size=11), borderwidth=0),
plot_bgcolor="white",
)
# Save the figure as JSON
avg_copy_numb_empo3_json = os.path.join(
avg_copy_numb_dir, "2_average_copy_number_emp_ontology_level3.json"
)
fig.write_json(avg_copy_numb_empo3_json)
# Save the figure as PNG
# avg_copy_numb_empo3_png = os.path.join(avg_copy_numb_dir, "3_avg_copy_numb_empo3.png")
# fig.write_image(avg_copy_numb_empo3_png, scale=2)
# Show the figure
fig.show(renderer="png")
2.3. Nestedness#
2.3.1. Load the data and create section and subsection directories#
Show code cell content
Hide code cell content
# Create the output directory for the metagenomics section and nestedness subsection
nestedness_dir = os.path.join(base_output_dir, "2_Metagenomics/3_nestedness")
os.makedirs(nestedness_dir, exist_ok=True)
# Load data
nest_phylum_allsamples = "https://raw.githubusercontent.com//biocore/emp/master/data/nestedness/nest_phylum_allsamples.csv"
nest_phylum_allsamples_df = pd.read_csv(nest_phylum_allsamples)
nest_phylum_animalsamples = "https://raw.githubusercontent.com//biocore/emp/master/data/nestedness/nest_phylum_Animal.csv"
nest_phylum_animalsamples_df = pd.read_csv(nest_phylum_animalsamples)
nest_phylum_plantsamples = "https://raw.githubusercontent.com//biocore/emp/master/data/nestedness/nest_phylum_Plant.csv"
nest_phylum_plantsamples_df = pd.read_csv(nest_phylum_plantsamples)
nest_phylum_nonsalinesamples = "https://raw.githubusercontent.com//biocore/emp/master/data/nestedness/nest_phylum_Non-saline.csv"
nest_phylum_nonsalinesamples_df = pd.read_csv(nest_phylum_nonsalinesamples)
2.3.2. Random subset of the nestedness dataframe#
Show code cell content
Hide code cell content
# Obtain a randome sample of the nestedness df for all samples
sample_nest_phylum_allsamples_df = nest_phylum_allsamples_df.sample(
100, random_state=42
)
# Export the sample df as a CSV file
sample_nest_phylum_allsamples_df.to_csv(
f"{nestedness_dir}/1_nestedness_random_subset.csv"
)
2.3.3. Interactive plots with plotly#
Show code cell source
Hide code cell source
# Determine axis limits
xmax = nest_phylum_allsamples_df.SAMPLE_RANK.max()
ymax = nest_phylum_allsamples_df.OBSERVATION_RANK.max()
# Get colors for each empo_3 category
nest_phylum_allsamples_df["color"] = nest_phylum_allsamples_df["empo_3"].apply(
get_empo_cat_color
)
# Create the scatter plot
fig = px.scatter(
nest_phylum_allsamples_df,
x="SAMPLE_RANK",
y="OBSERVATION_RANK",
color="empo_3",
color_discrete_map={
empo: get_empo_cat_color(empo)
for empo in nest_phylum_allsamples_df["empo_3"].unique()
},
labels={
"SAMPLE_RANK": "All samples (sorted by richness)",
"OBSERVATION_RANK": "Phyla (sorted by prevalence)",
},
template="plotly_white",
)
# Customize layout
fig.update_traces(marker=dict(symbol="line-ns-open", size=10)) # "|" marker equivalent
fig.update_layout(
width=1200, # Increase width for a wider plot
height=600, # Adjust height if needed
xaxis=dict(range=[0, xmax + 1], title_font=dict(size=20), tickfont=dict(size=18)),
yaxis=dict(range=[0, ymax + 0.8], title_font=dict(size=20), tickfont=dict(size=18)),
showlegend=False,
)
# Save the figure as an interactive HTML file
os.makedirs(nestedness_dir, exist_ok=True)
nest_phylum_allsamples_path = os.path.join(nestedness_dir, "2_all_samples.json")
fig.write_json(nest_phylum_allsamples_path)
# Show the figure
fig.show(renderer="png")
Show code cell content
Hide code cell content
# Determine axis limits
xmax = nest_phylum_plantsamples_df.SAMPLE_RANK.max()
ymax = nest_phylum_plantsamples_df.OBSERVATION_RANK.max()
# Get colors for each empo_3 category
nest_phylum_plantsamples_df["color"] = nest_phylum_plantsamples_df["empo_3"].apply(
get_empo_cat_color
)
# Create the scatter plot
fig = px.scatter(
nest_phylum_plantsamples_df,
x="SAMPLE_RANK",
y="OBSERVATION_RANK",
color="empo_3",
color_discrete_map={
empo: get_empo_cat_color(empo)
for empo in nest_phylum_plantsamples_df["empo_3"].unique()
},
labels={
"SAMPLE_RANK": "Plant samples (sorted by richness)",
"OBSERVATION_RANK": "Phyla (sorted by prevalence)",
},
template="plotly_white",
)
# Customize layout
fig.update_traces(marker=dict(symbol="line-ns-open", size=10)) # "|" marker equivalent
fig.update_layout(
width=1200, # Increase width for a wider plot
height=600, # Adjust height if needed
xaxis=dict(range=[0, xmax + 1], title_font=dict(size=20), tickfont=dict(size=18)),
yaxis=dict(range=[0, ymax + 0.8], title_font=dict(size=20), tickfont=dict(size=18)),
legend=dict(
orientation="h", # Horizontal legend
yanchor="top",
y=-0.2, # Moves the legend below the x-axis
xanchor="center",
x=0.5, # Centers the legend
font=dict(size=16),
),
legend_title_text="",
)
# Save the figure as an interactive HTML file
os.makedirs(nestedness_dir, exist_ok=True)
nest_phylum_plantsamples_path = os.path.join(nestedness_dir, "3_plant_samples.json")
fig.write_json(nest_phylum_plantsamples_path)
# Show the figure
fig.show(renderer="png")
2.3.4. Static plots with matplotlib#
Show code cell source
Hide code cell source
# Determine axis limits
xmax = nest_phylum_animalsamples_df.SAMPLE_RANK.max()
ymax = nest_phylum_animalsamples_df.OBSERVATION_RANK.max()
# Create the figure and axis
fig, ax = plt.subplots(figsize=(500 / 30, 80 / 12.7)) # Adjust size as needed
# Store legend handles and labels
legend_handles = []
legend_labels = []
# Plot data
for empo3 in np.sort(nest_phylum_animalsamples_df.empo_3.unique()):
color = get_empo_cat_color(empo3)
scatter = ax.scatter(
nest_phylum_animalsamples_df[
nest_phylum_animalsamples_df.empo_3 == empo3
].SAMPLE_RANK,
nest_phylum_animalsamples_df[
nest_phylum_animalsamples_df.empo_3 == empo3
].OBSERVATION_RANK,
marker="|",
linewidths=2,
label=empo3,
color=color,
)
legend_handles.append(scatter)
legend_labels.append(empo3)
# Customize labels and appearance
ax.set_xlabel("Animal samples (sorted by richness)", fontsize=20)
ax.set_ylabel("Phyla (sorted by prevalence)", fontsize=20)
ax.tick_params(axis="both", which="major", labelsize=18)
# Add legend
ax.legend(
handles=legend_handles,
labels=legend_labels,
loc="upper center",
bbox_to_anchor=(0.5, -0.2), # Moves legend below x-axis
ncol=3,
fontsize=16,
frameon=False,
scatterpoints=1,
handletextpad=0.5,
)
# Increase space at the bottom so the legend is not cut off
plt.subplots_adjust(bottom=0.25)
ax.set_xlim([0, xmax + 1])
ax.set_ylim([0, ymax + 0.8])
plt.tight_layout()
fig.patch.set_alpha(0.0)
# Save the figure
os.makedirs(nestedness_dir, exist_ok=True)
nest_phylum_animalsamples_path = os.path.join(nestedness_dir, "4_animal_samples.png")
plt.savefig(nest_phylum_animalsamples_path, dpi=300, bbox_inches="tight")
Show code cell source
Hide code cell source
# Determine axis limits
xmax = nest_phylum_nonsalinesamples_df.SAMPLE_RANK.max()
ymax = nest_phylum_nonsalinesamples_df.OBSERVATION_RANK.max()
# Create the figure and axis
fig, ax = plt.subplots(figsize=(500 / 30, 80 / 12.7)) # Adjust size as needed
# Store legend handles and labels
legend_handles = []
legend_labels = []
# Plot data
for empo3 in np.sort(nest_phylum_nonsalinesamples_df.empo_3.unique()):
color = get_empo_cat_color(empo3)
scatter = ax.scatter(
nest_phylum_nonsalinesamples_df[
nest_phylum_nonsalinesamples_df.empo_3 == empo3
].SAMPLE_RANK,
nest_phylum_nonsalinesamples_df[
nest_phylum_nonsalinesamples_df.empo_3 == empo3
].OBSERVATION_RANK,
marker="|",
linewidths=2,
label=empo3,
color=color,
)
legend_handles.append(scatter)
legend_labels.append(empo3)
# Customize labels and appearance
ax.set_xlabel("Non saline samples (sorted by richness)", fontsize=20)
ax.set_ylabel("Phyla (sorted by prevalence)", fontsize=20)
ax.tick_params(axis="both", which="major", labelsize=18)
# Add legend
ax.legend(
handles=legend_handles,
labels=legend_labels,
loc="upper center",
bbox_to_anchor=(0.5, -0.2), # Moves legend below x-axis
ncol=3,
fontsize=16,
frameon=False,
scatterpoints=1,
handletextpad=0.5,
)
# Increase space at the bottom so the legend is not cut off
plt.subplots_adjust(bottom=0.25)
ax.set_xlim([0, xmax + 1])
ax.set_ylim([0, ymax + 0.8])
plt.tight_layout()
fig.patch.set_alpha(0.0)
# Save the figure
os.makedirs(nestedness_dir, exist_ok=True)
nest_phylum_nonsalinesamples_path = os.path.join(
nestedness_dir, "5_non_saline_samples.png"
)
plt.savefig(nest_phylum_nonsalinesamples_path, dpi=300, bbox_inches="tight")
3. Network analysis#
3.1. Phyla association networks#
3.1.1 Load data and create section and subsection directories#
Show code cell content
Hide code cell content
# Create the output directory for the network analysis section and microbial networks subsection
network_dir = os.path.join(
base_output_dir, "3_Network_analysis/1_phyla_association_networks"
)
os.makedirs(network_dir, exist_ok=True)
# Load OTU counts table
otu_counts = "https://raw.githubusercontent.com//biocore/emp/master/data/biom-tables/emp_deblur_100bp.subset_2k.rare_5000.biom"
# Download the file and save it as a binary file
response = requests.get(otu_counts)
with open(
"example_data/Earth_microbiome_vuegen_demo_notebook/emp_deblur_100bp.subset_2k.rare_5000.biom",
"wb",
) as f:
f.write(response.content)
# Load the BIOM file and convert it to a DataFrame
otu_counts_table = biom.load_table(
"example_data/Earth_microbiome_vuegen_demo_notebook/emp_deblur_100bp.subset_2k.rare_5000.biom"
)
Show code cell content
Hide code cell content
# Collapse the table to the phylum level
phylum_idx = 1
collapse_f = lambda id_, md: "; ".join(md["taxonomy"][: phylum_idx + 1])
phyla_table = otu_counts_table.collapse(collapse_f, axis="observation")
# Convert the collapsed table to a DataFrame
phyla_counts_df = phyla_table.to_dataframe()
3.1.2. Clean data and export a subset of the counts table#
Show code cell content
Hide code cell content
# Clean the index (which contains Phylum names) by removing unnecessary parts
phyla_counts_df.index = (
phyla_counts_df.index.str.split(";").str[-1].str.replace("p__", "", regex=False)
)
# Remove special characters like [] and unnecessary spaces
phyla_counts_df.index = (
phyla_counts_df.index.str.replace("[", "", regex=False)
.str.replace("]", "", regex=False)
.str.strip()
)
# Remove rows where the index only has 'k__' and 'Unclassified'
phyla_counts_df = phyla_counts_df[~(phyla_counts_df.index == "Unclassified")]
phyla_counts_df = phyla_counts_df[~phyla_counts_df.index.str.contains("k__")]
# Remove duplicaye rows
phyla_counts_df = phyla_counts_df[~phyla_counts_df.index.duplicated(keep="first")]
Show code cell content
Hide code cell content
# Obtain a randome sample of the columns of the phyla counts df
sample_phyla_counts_df = phyla_counts_df.sample(50, axis=1)
# Export the sample df as a CSV file
sample_phyla_counts_df.to_csv(f"{network_dir}/1_phyla_counts_subset.csv")
3.1.3. Phyla association network edge list#
Show code cell content
Hide code cell content
# Compute correlation matrix with absolute values
phyla_corr_matrix = phyla_counts_df.T.corr().abs()
# Create a binary adjacency matrix using a threshold
threshold = 0.5
phyla_corr_matrix_binary = phyla_corr_matrix > threshold
# Set diagonal values to 0
np.fill_diagonal(phyla_corr_matrix_binary.values, 0)
Show code cell content
Hide code cell content
# Create a graph from the binary matrix
G = nx.Graph()
# Add nodes (Phyla)
phyla = phyla_corr_matrix_binary.index.tolist()
G.add_nodes_from(phyla)
# Add edges where binary_matrix == 1
for i in range(len(phyla)):
for j in range(i + 1, len(phyla)): # Avoid duplicate edges
if phyla_corr_matrix_binary.iloc[i, j] == 1:
G.add_edge(phyla[i], phyla[j])
# Remove singleton nodes (nodes with no edges)
G.remove_nodes_from(list(nx.isolates(G)))
# Export network as an edge list in CSV format, the "edge_list" word should be in the file name to be
# recognized as an edge list file
edge_list = nx.to_pandas_edgelist(G)
edge_list.to_csv(
os.path.join(
network_dir, "2_phyla_correlation_network_with_0.5_threshold_edgelist.csv"
),
index=False,
)
3.1.4. Phyla association network static plot#
Show code cell source
Hide code cell source
# Draw the network
plt.figure(figsize=(8, 6))
pos = nx.kamada_kawai_layout(G) # Layout for better visualization
nx.draw(
G,
pos,
with_labels=True,
node_size=500,
node_color="lightblue",
edgecolors="black",
linewidths=0.3,
font_size=10,
)
# Export the figure as a PNG file
network_path = os.path.join(
network_dir, "3_phyla_correlation_network_with_0.5_threshold.png"
)
plt.savefig(network_path, dpi=300, bbox_inches="tight")
4. Report generation#
To generate different report types, just modify the report_type variable. The available types are:
streamlit
html
pdf
docx
odt
revealjs
pptx
jupyter
4.1. Streamlit report#
To launch the Streamlit web application from Colab, open the generated URL and copy the localtunnel entry point IP into the corresponding field on the opened page. Once submited, you will be redirected to your Streamlit web application.
# Generate the report
report_type = "streamlit"
_ = report_generator.get_report(
dir_path=base_output_dir, report_type=report_type, logger=None
)
Show code cell output
Hide code cell output
[2026-02-10 13:00:21,877] vuegen: INFO - Path to log file: logs/2026-02-10_13-00-21_report.log
[2026-02-10 13:00:21,920] vuegen: ERROR - Unsupported file extension: .yaml. Skipping file: example_data/Earth_microbiome_vuegen_demo_notebook/Earth_microbiome_vuegen_demo_notebook_config.yaml
[2026-02-10 13:00:21,921] vuegen: ERROR - Unsupported file extension: .biom. Skipping file: example_data/Earth_microbiome_vuegen_demo_notebook/emp_deblur_100bp.subset_2k.rare_5000.biom
[2026-02-10 13:00:21,930] vuegen: INFO - Configuration file generated at earth_microbiome_vuegen_demo_notebook_config.yaml
[2026-02-10 13:00:21,942] vuegen: INFO - Report 'Earth Microbiome Vuegen Demo Notebook' initialized with 3 sections.
[2026-02-10 13:00:21,943] vuegen: INFO - running in a normal Python process
[2026-02-10 13:00:21,944] vuegen: DEBUG - Generating 'streamlit' report in directory: 'streamlit_report/sections'
[2026-02-10 13:00:21,944] vuegen: INFO - Output directory already existed: 'streamlit_report/sections'
[2026-02-10 13:00:21,945] vuegen: INFO - Output directory for static content already existed: 'streamlit_report/static'
[2026-02-10 13:00:21,946] vuegen: DEBUG - Processing app navigation code.
[2026-02-10 13:00:21,947] vuegen: DEBUG - Processing home section.
[2026-02-10 13:00:21,948] vuegen: DEBUG - Home directory already existed: streamlit_report/sections/Home
[2026-02-10 13:00:21,949] vuegen: INFO - Home page content written to 'streamlit_report/sections/Home/Homepage.py'.
[2026-02-10 13:00:21,949] vuegen: INFO - Home page added to the report manager content.
[2026-02-10 13:00:21,951] vuegen: DEBUG - Created section directory: streamlit_report/sections/Exploratory_Data_Analysis
[2026-02-10 13:00:21,951] vuegen: DEBUG - Created section directory: streamlit_report/sections/Metagenomics
[2026-02-10 13:00:21,952] vuegen: DEBUG - Created section directory: streamlit_report/sections/Network_Analysis
[2026-02-10 13:00:21,953] vuegen: INFO - Created app navigation script: report_manager.py
[2026-02-10 13:00:21,954] vuegen: INFO - Starting to generate sections for the report.
[2026-02-10 13:00:21,955] vuegen: DEBUG - Processing section '1': 'Exploratory Data Analysis' - 1 subsection(s)
[2026-02-10 13:00:21,956] vuegen: DEBUG - Processing subsection '1': 'Sample Exploration' - 4 component(s)
[2026-02-10 13:00:21,957] vuegen: INFO - Successfully generated content for DataFrame: 'Metadata Random Subset'
[2026-02-10 13:00:21,958] vuegen: INFO - Successfully generated content for plot '1': 'Animal Samples Map'
[2026-02-10 13:00:21,960] vuegen: INFO - Successfully generated content for plot '2': 'Plant Samples Map'
[2026-02-10 13:00:21,961] vuegen: INFO - Successfully generated content for plot '3': 'Saline Samples Map'
[2026-02-10 13:00:21,961] vuegen: INFO - Generated content and imports for subsection: 'Sample Exploration'
[2026-02-10 13:00:21,962] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Exploratory_Data_Analysis/Sample_Exploration.py'
[2026-02-10 13:00:21,963] vuegen: DEBUG - Processing section '2': 'Metagenomics' - 3 subsection(s)
[2026-02-10 13:00:21,964] vuegen: DEBUG - Processing subsection '2': 'Alpha Diversity' - 2 component(s)
[2026-02-10 13:00:21,965] vuegen: INFO - Successfully generated content for plot '4': 'Alpha Diversity Host Associated Samples'
[2026-02-10 13:00:21,966] vuegen: INFO - Successfully generated content for plot '5': 'Alpha Diversity Free Living Samples'
[2026-02-10 13:00:21,967] vuegen: INFO - Generated content and imports for subsection: 'Alpha Diversity'
[2026-02-10 13:00:21,968] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Metagenomics/Alpha_Diversity.py'
[2026-02-10 13:00:21,969] vuegen: DEBUG - Processing subsection '3': 'Average Copy Number' - 2 component(s)
[2026-02-10 13:00:21,970] vuegen: INFO - Successfully generated content for plot '6': 'Average Copy Number Emp Ontology Level2'
[2026-02-10 13:00:21,971] vuegen: INFO - Successfully generated content for plot '7': 'Average Copy Number Emp Ontology Level3'
[2026-02-10 13:00:21,971] vuegen: INFO - Generated content and imports for subsection: 'Average Copy Number'
[2026-02-10 13:00:21,973] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Metagenomics/Average_Copy_Number.py'
[2026-02-10 13:00:21,973] vuegen: DEBUG - Processing subsection '4': 'Nestedness' - 5 component(s)
[2026-02-10 13:00:21,975] vuegen: INFO - Successfully generated content for DataFrame: 'Nestedness Random Subset'
[2026-02-10 13:00:21,976] vuegen: INFO - Successfully generated content for plot '8': 'All Samples'
[2026-02-10 13:00:21,977] vuegen: INFO - Successfully generated content for plot '9': 'Plant Samples'
[2026-02-10 13:00:21,978] vuegen: INFO - Successfully generated content for plot '10': 'Animal Samples'
[2026-02-10 13:00:21,980] vuegen: INFO - Successfully generated content for plot '11': 'Non Saline Samples'
[2026-02-10 13:00:21,980] vuegen: INFO - Generated content and imports for subsection: 'Nestedness'
[2026-02-10 13:00:21,981] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Metagenomics/Nestedness.py'
[2026-02-10 13:00:21,982] vuegen: DEBUG - Processing section '3': 'Network Analysis' - 1 subsection(s)
[2026-02-10 13:00:21,982] vuegen: DEBUG - Processing subsection '5': 'Phyla Association Networks' - 3 component(s)
[2026-02-10 13:00:21,984] vuegen: INFO - Successfully generated content for DataFrame: 'Phyla Counts Subset'
[2026-02-10 13:00:21,988] vuegen: INFO - Successfully read network from file: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/example_data/Earth_microbiome_vuegen_demo_notebook/3_Network_analysis/1_phyla_association_networks/2_phyla_correlation_network_with_0.5_threshold_edgelist.csv.
[2026-02-10 13:00:21,988] vuegen: DEBUG - Try to save network as PyVis network: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/streamlit_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:22,002] vuegen: INFO - PyVis network created and saved as: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/streamlit_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:22,005] vuegen: INFO - Successfully generated content for plot '12': 'Phyla Correlation Network With 0.5 Threshold Edgelist'
[2026-02-10 13:00:22,006] vuegen: INFO - Successfully generated content for plot '13': 'Phyla Correlation Network With 0.5 Threshold'
[2026-02-10 13:00:22,006] vuegen: INFO - Generated content and imports for subsection: 'Phyla Association Networks'
[2026-02-10 13:00:22,008] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Network_Analysis/Phyla_Association_Networks.py'
[2026-02-10 13:00:22,009] vuegen: INFO - All the scripts to build the Streamlit app are available at streamlit_report/sections
[2026-02-10 13:00:22,009] vuegen: INFO - To run the Streamlit app, use the following command:
[2026-02-10 13:00:22,010] vuegen: INFO - streamlit run streamlit_report/sections/report_manager.py
All the scripts to build the Streamlit app are available at: streamlit_report/sections
To run the Streamlit app, use the following command:
streamlit run streamlit_report/sections/report_manager.py
run_streamlit = False
# run_streamlit = True # uncomment line to run the streamlit report
# Launch the Streamlit report depneding on the platform
if not IN_COLAB and run_streamlit:
!streamlit run streamlit_report/sections/report_manager.py
elif run_streamlit:
# see: https://discuss.streamlit.io/t/how-to-launch-streamlit-app-from-google-colab-notebook/42399
print(
"Password/Enpoint IP for localtunnel is:",
urllib.request.urlopen("https://ipv4.icanhazip.com")
.read()
.decode("utf8")
.strip("\n"),
)
# Run the Streamlit app in the background
!streamlit run streamlit_report/sections/report_manager.py --server.address=localhost &>/content/logs.txt &
# Expose the Streamlit app on port 8501
!npx localtunnel --port 8501 --subdomain vuegen-demo
else:
print("Streamlit report not executed, set run_streamlit to True to run the report")
Show code cell output
Hide code cell output
Streamlit report not executed, set run_streamlit to True to run the report
4.2. HTML and other report types#
# Generate the report
report_type = "html"
report_dir, config_path = report_generator.get_report(
dir_path=base_output_dir, report_type=report_type, logger=None
)
Show code cell output
Hide code cell output
[2026-02-10 13:00:22,026] vuegen: INFO - Path to log file: logs/2026-02-10_13-00-22_report.log
[2026-02-10 13:00:22,071] vuegen: ERROR - Unsupported file extension: .yaml. Skipping file: example_data/Earth_microbiome_vuegen_demo_notebook/Earth_microbiome_vuegen_demo_notebook_config.yaml
[2026-02-10 13:00:22,072] vuegen: ERROR - Unsupported file extension: .biom. Skipping file: example_data/Earth_microbiome_vuegen_demo_notebook/emp_deblur_100bp.subset_2k.rare_5000.biom
[2026-02-10 13:00:22,080] vuegen: INFO - Configuration file generated at earth_microbiome_vuegen_demo_notebook_config.yaml
[2026-02-10 13:00:22,092] vuegen: INFO - Report 'Earth Microbiome Vuegen Demo Notebook' initialized with 3 sections.
[2026-02-10 13:00:22,093] vuegen: INFO - running in a normal Python process
[2026-02-10 13:00:22,094] vuegen: DEBUG - env_vars (QuartoReport): environ({'READTHEDOCS_VIRTUALENV_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable', 'READTHEDOCS_CANONICAL_URL': 'https://vuegen.readthedocs.io/en/stable/', 'HOSTNAME': 'build-31350056-project-1135329-vuegen', 'READTHEDOCS_GIT_CLONE_URL': 'https://github.com/Multiomics-Analytics-Group/vuegen.git', 'HOME': '/home/docs', 'NO_COLOR': '1', 'READTHEDOCS': 'True', 'READTHEDOCS_PRODUCTION_DOMAIN': 'app.readthedocs.org', 'READTHEDOCS_REPOSITORY_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable', 'READTHEDOCS_PROJECT': 'vuegen', 'READTHEDOCS_OUTPUT': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/_readthedocs/', 'PATH': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/bin:/home/docs/.asdf/shims:/home/docs/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'READTHEDOCS_VERSION_TYPE': 'tag', 'LANG': 'C.UTF-8', 'READTHEDOCS_LANGUAGE': 'en', 'DEBIAN_FRONTEND': 'noninteractive', 'READTHEDOCS_GIT_COMMIT_HASH': '5258cc795a2ea011d75b5c1e450ade2f7b59c8a5', 'READTHEDOCS_VERSION_NAME': 'stable', 'READTHEDOCS_VERSION': 'stable', 'PWD': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs', 'READTHEDOCS_GIT_IDENTIFIER': 'v0.6.0', 'DOCUTILSCONFIG': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/docutils.conf', 'MPLBACKEND': 'Agg', 'JPY_PARENT_PID': '698', 'PYDEVD_USE_FRAME_EVAL': 'NO', 'TERM': 'xterm-color', 'CLICOLOR': '1', 'FORCE_COLOR': '1', 'CLICOLOR_FORCE': '1', 'PAGER': 'cat', 'GIT_PAGER': 'cat'})
[2026-02-10 13:00:22,095] vuegen: DEBUG - PATH: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/bin:/home/docs/.asdf/shims:/home/docs/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[2026-02-10 13:00:22,096] vuegen: DEBUG - sys.path: ['/home/docs/.asdf/installs/python/3.11.12/lib/python311.zip', '/home/docs/.asdf/installs/python/3.11.12/lib/python3.11', '/home/docs/.asdf/installs/python/3.11.12/lib/python3.11/lib-dynload', '', '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages']
[2026-02-10 13:00:22,096] vuegen: DEBUG - Generating 'html' report in directory: '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report'
[2026-02-10 13:00:22,097] vuegen: DEBUG - Output directory already existed: '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report'
[2026-02-10 13:00:22,097] vuegen: INFO - Output directory for static content already existed: 'quarto_report/static'
[2026-02-10 13:00:22,099] vuegen: INFO - Starting to generate sections for the report.
[2026-02-10 13:00:22,100] vuegen: DEBUG - Processing section: 'Exploratory Data Analysis' - 1 subsection(s)
[2026-02-10 13:00:22,101] vuegen: DEBUG - Processing subsection: 'Sample Exploration' - 4 component(s)
[2026-02-10 13:00:22,101] vuegen: DEBUG - component_imports: ['import pandas as pd', 'from itables import show, init_notebook_mode', 'init_notebook_mode(all_interactive=True)']
[2026-02-10 13:00:22,103] vuegen: INFO - Successfully generated content for DataFrame: 'Metadata Random Subset'
[2026-02-10 13:00:22,103] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:22,104] vuegen: INFO - Successfully generated content for plot: 'Animal Samples Map'
[2026-02-10 13:00:22,105] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:22,106] vuegen: INFO - Successfully generated content for plot: 'Plant Samples Map'
[2026-02-10 13:00:22,107] vuegen: DEBUG - component_imports: ['import altair as alt', 'import requests', 'import json']
[2026-02-10 13:00:22,108] vuegen: INFO - Successfully generated content for plot: 'Saline Samples Map'
[2026-02-10 13:00:22,109] vuegen: INFO - Generated content and imports for subsection: 'Sample Exploration'
[2026-02-10 13:00:22,110] vuegen: DEBUG - Processing section: 'Metagenomics' - 3 subsection(s)
[2026-02-10 13:00:22,111] vuegen: DEBUG - Processing subsection: 'Alpha Diversity' - 2 component(s)
[2026-02-10 13:00:22,111] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:22,112] vuegen: INFO - Successfully generated content for plot: 'Alpha Diversity Host Associated Samples'
[2026-02-10 13:00:22,113] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:22,114] vuegen: INFO - Successfully generated content for plot: 'Alpha Diversity Free Living Samples'
[2026-02-10 13:00:22,115] vuegen: INFO - Generated content and imports for subsection: 'Alpha Diversity'
[2026-02-10 13:00:22,116] vuegen: DEBUG - Processing subsection: 'Average Copy Number' - 2 component(s)
[2026-02-10 13:00:22,117] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:22,118] vuegen: INFO - Successfully generated content for plot: 'Average Copy Number Emp Ontology Level2'
[2026-02-10 13:00:22,118] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:22,120] vuegen: INFO - Successfully generated content for plot: 'Average Copy Number Emp Ontology Level3'
[2026-02-10 13:00:22,120] vuegen: INFO - Generated content and imports for subsection: 'Average Copy Number'
[2026-02-10 13:00:22,121] vuegen: DEBUG - Processing subsection: 'Nestedness' - 5 component(s)
[2026-02-10 13:00:22,122] vuegen: DEBUG - component_imports: ['import pandas as pd', 'from itables import show, init_notebook_mode', 'init_notebook_mode(all_interactive=True)']
[2026-02-10 13:00:22,123] vuegen: INFO - Successfully generated content for DataFrame: 'Nestedness Random Subset'
[2026-02-10 13:00:22,124] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:22,125] vuegen: INFO - Successfully generated content for plot: 'All Samples'
[2026-02-10 13:00:22,126] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:22,127] vuegen: INFO - Successfully generated content for plot: 'Plant Samples'
[2026-02-10 13:00:22,127] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:22,129] vuegen: INFO - Successfully generated content for plot: 'Animal Samples'
[2026-02-10 13:00:22,129] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:22,131] vuegen: INFO - Successfully generated content for plot: 'Non Saline Samples'
[2026-02-10 13:00:22,131] vuegen: INFO - Generated content and imports for subsection: 'Nestedness'
[2026-02-10 13:00:22,132] vuegen: DEBUG - Processing section: 'Network Analysis' - 1 subsection(s)
[2026-02-10 13:00:22,133] vuegen: DEBUG - Processing subsection: 'Phyla Association Networks' - 3 component(s)
[2026-02-10 13:00:22,134] vuegen: DEBUG - component_imports: ['import pandas as pd', 'from itables import show, init_notebook_mode', 'init_notebook_mode(all_interactive=True)']
[2026-02-10 13:00:22,135] vuegen: INFO - Successfully generated content for DataFrame: 'Phyla Counts Subset'
[2026-02-10 13:00:22,135] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:22,139] vuegen: INFO - Successfully read network from file: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/example_data/Earth_microbiome_vuegen_demo_notebook/3_Network_analysis/1_phyla_association_networks/2_phyla_correlation_network_with_0.5_threshold_edgelist.csv.
[2026-02-10 13:00:22,140] vuegen: DEBUG - Try to save network as PyVis network: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:22,154] vuegen: INFO - PyVis network created and saved as: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:22,155] vuegen: INFO - Successfully generated content for plot: 'Phyla Correlation Network With 0.5 Threshold Edgelist'
[2026-02-10 13:00:22,156] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:22,157] vuegen: INFO - Successfully generated content for plot: 'Phyla Correlation Network With 0.5 Threshold'
[2026-02-10 13:00:22,158] vuegen: INFO - Generated content and imports for subsection: 'Phyla Association Networks'
[2026-02-10 13:00:22,159] vuegen: INFO - Created qmd script to render the app: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/quarto_report.qmd
[2026-02-10 13:00:22,160] vuegen: INFO - Running 'Earth Microbiome Vuegen Demo Notebook' 'html' report with ['quarto', 'render', '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/quarto_report.qmd']
[2026-02-10 13:00:43,668] vuegen: INFO - 'Earth Microbiome Vuegen Demo Notebook' 'html' report rendered
Starting python3 kernel...
Done
Executing 'quarto_report.quarto_ipynb'
Cell 1/10: 'Imports'......................................
Done
Cell 2/10: 'Metadata Random Subset 4'.....................Done
Cell 3/10: 'Plant Samples Map 15'.........................Done
Cell 4/10: 'Saline Samples Map 16'........................
Done
Cell 5/10: 'Alpha Diversity Free Living Samples 18'.......Done
Cell 6/10: 'Average Copy Number Emp Ontology Level3 20'...
Done
Cell 7/10: 'Nestedness Random Subset 5'...................Done
Cell 8/10: 'All Samples 21'...............................
Done
Cell 9/10: 'Plant Samples 22'.............................Done
Cell 10/10: 'Phyla Counts Subset 6'........................Done
pandoc
to: html
output-file: quarto_report.html
standalone: true
self-contained: true
section-divs: true
html-math-method: mathjax
wrap: none
default-image-extension: png
toc: true
toc-depth: 3
variables: {}
metadata
document-css: false
link-citations: true
date-format: long
lang: en
title: Earth Microbiome Vuegen Demo Notebook
jupyter: python3
toc-location: left
page-layout: full
Output created: quarto_report.html
5. Extending the report#
Now, we will extend the report by modifying the configuration file to include a logo and graphica abstract in the main page, a descritption for a section and a subsection, and a new plot from a url. We are modifying this file from the notebook, but it is also possible to do it directly in the configuration file with a text editor.
5.1. Adding a logo and graphical abstract#
Show code cell content
Hide code cell content
empo_logo_path = (
"https://raw.githubusercontent.com/ElDeveloper/cogs220/master/emp-logo.svg"
)
# Load the YAML file
config = load_yaml_config(config_path)
# Update the logo and graphical abstract with the URL
config["report"].update({"logo": empo_logo_path, "graphical_abstract": empo_logo_path})
5.2. Including a description for a section and a subsection#
Show code cell content
Hide code cell content
# Update the description for the EDA section
for section in config["sections"]:
if section["title"] == "Exploratory Data Analysis":
section["description"] = (
"This section contains the exploratory data analysis of the Earth Microbiome Project (EMP) dataset."
)
# Update the description for the alpha diversity subsection from the Metagenomics section
for section in config["sections"]:
if section["title"] == "Metagenomics":
for subsection in section["subsections"]:
if subsection["title"] == "Alpha Diversity":
subsection["description"] = (
"This subsection contains the alpha diversity analysis of the EMP dataset."
)
5.3. Adding a new plot from a url#
Show code cell content
Hide code cell content
# Define new plot with a URL as the file path
chem_prop_plot = {
"title": "Physicochemical properties of the EMP samples",
"file_path": "https://raw.githubusercontent.com/biocore/emp/master/methods/images/figureED1_physicochemical.png",
"description": "",
"caption": "Pairwise scatter plots of available physicochemical metadat are shown for temperature, salinity, oxygen, and pH, and for phosphate, nitrate, and ammonium",
"component_type": "plot",
"plot_type": "static",
}
# Add the plot to the Sample Provenance subsection in the EDA section
for section in config["sections"]:
if section["title"] == "Exploratory Data Analysis":
for subsection in section["subsections"]:
if subsection["title"] == "Sample Exploration":
subsection["components"].append(chem_prop_plot)
5.4. Creating a new subsection with a plot from a url#
Show code cell content
Hide code cell content
# Define new plot with a URL as the file path
specif_seq_plot = {
"title": "Specificity of sequences and higher taxonomic groups for environment",
"file_path": "https://raw.githubusercontent.com/biocore/emp/master/methods/images/figure4_entropy.png",
"description": "",
"caption": "a) Environment distribution in all genera and 400 randomly chosen tag sequence. b) and c) Shannon entropy within each taxonomic group.",
"component_type": "plot",
"plot_type": "static",
}
# Define the new subsection for the Shannon entropy analysis
entropy_subsection = {
"title": "Shanon entropy analysis",
"description": "This subsection contains the Shannon entropy analysis of the EMP dataset.",
"components": [specif_seq_plot],
}
# Add the new subsection to the Metagenomics section
for section in config["sections"]:
if section["title"] == "Metagenomics":
section["subsections"].append(entropy_subsection)
# Save the modified YAML file
with open(config_path, "w") as file:
yaml.dump(config, file, default_flow_style=False)
5.5. Stremlit report with the extended configuration file#
To launch the Streamlit web application from Colab, open the generated URL and copy the localtunnel entry point IP into the corresponding field on the opened page. Once submited, you will be redirected to your Streamlit web application.
# Test the changes by generarating the report from the modified YAML file
report_type = "streamlit"
_ = report_generator.get_report(
config_path=config_path, report_type=report_type, logger=None
)
Show code cell output
Hide code cell output
[2026-02-10 13:00:43,721] vuegen: INFO - Path to log file: logs/2026-02-10_13-00-43_report.log
[2026-02-10 13:00:43,734] vuegen: INFO - Report 'Earth Microbiome Vuegen Demo Notebook' initialized with 3 sections.
[2026-02-10 13:00:43,735] vuegen: INFO - running in a normal Python process
[2026-02-10 13:00:43,736] vuegen: DEBUG - Generating 'streamlit' report in directory: 'streamlit_report/sections'
[2026-02-10 13:00:43,737] vuegen: INFO - Output directory already existed: 'streamlit_report/sections'
[2026-02-10 13:00:43,737] vuegen: INFO - Output directory for static content already existed: 'streamlit_report/static'
[2026-02-10 13:00:43,739] vuegen: DEBUG - Processing app navigation code.
[2026-02-10 13:00:43,739] vuegen: DEBUG - Processing home section.
[2026-02-10 13:00:43,740] vuegen: DEBUG - Home directory already existed: streamlit_report/sections/Home
[2026-02-10 13:00:43,741] vuegen: INFO - Home page content written to 'streamlit_report/sections/Home/Homepage.py'.
[2026-02-10 13:00:43,742] vuegen: INFO - Home page added to the report manager content.
[2026-02-10 13:00:43,743] vuegen: DEBUG - Section directory already existed: streamlit_report/sections/Exploratory_Data_Analysis
[2026-02-10 13:00:43,744] vuegen: DEBUG - Section directory already existed: streamlit_report/sections/Metagenomics
[2026-02-10 13:00:43,745] vuegen: DEBUG - Section directory already existed: streamlit_report/sections/Network_Analysis
[2026-02-10 13:00:43,746] vuegen: INFO - Created app navigation script: report_manager.py
[2026-02-10 13:00:43,747] vuegen: INFO - Starting to generate sections for the report.
[2026-02-10 13:00:43,747] vuegen: DEBUG - Processing section '7': 'Exploratory Data Analysis' - 1 subsection(s)
[2026-02-10 13:00:43,748] vuegen: DEBUG - Processing subsection '11': 'Sample Exploration' - 5 component(s)
[2026-02-10 13:00:43,750] vuegen: INFO - Successfully generated content for DataFrame: 'Metadata Random Subset'
[2026-02-10 13:00:43,751] vuegen: INFO - Successfully generated content for plot '27': 'Animal Samples Map'
[2026-02-10 13:00:43,752] vuegen: INFO - Successfully generated content for plot '28': 'Plant Samples Map'
[2026-02-10 13:00:43,753] vuegen: INFO - Successfully generated content for plot '29': 'Saline Samples Map'
[2026-02-10 13:00:43,754] vuegen: INFO - Successfully generated content for plot '30': 'Physicochemical properties of the EMP samples'
[2026-02-10 13:00:43,755] vuegen: INFO - Generated content and imports for subsection: 'Sample Exploration'
[2026-02-10 13:00:43,756] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Exploratory_Data_Analysis/Sample_Exploration.py'
[2026-02-10 13:00:43,756] vuegen: DEBUG - Processing section '8': 'Metagenomics' - 4 subsection(s)
[2026-02-10 13:00:43,757] vuegen: DEBUG - Processing subsection '12': 'Alpha Diversity' - 2 component(s)
[2026-02-10 13:00:43,759] vuegen: INFO - Successfully generated content for plot '31': 'Alpha Diversity Host Associated Samples'
[2026-02-10 13:00:43,760] vuegen: INFO - Successfully generated content for plot '32': 'Alpha Diversity Free Living Samples'
[2026-02-10 13:00:43,760] vuegen: INFO - Generated content and imports for subsection: 'Alpha Diversity'
[2026-02-10 13:00:43,762] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Metagenomics/Alpha_Diversity.py'
[2026-02-10 13:00:43,762] vuegen: DEBUG - Processing subsection '13': 'Average Copy Number' - 2 component(s)
[2026-02-10 13:00:43,764] vuegen: INFO - Successfully generated content for plot '33': 'Average Copy Number Emp Ontology Level2'
[2026-02-10 13:00:43,765] vuegen: INFO - Successfully generated content for plot '34': 'Average Copy Number Emp Ontology Level3'
[2026-02-10 13:00:43,765] vuegen: INFO - Generated content and imports for subsection: 'Average Copy Number'
[2026-02-10 13:00:43,766] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Metagenomics/Average_Copy_Number.py'
[2026-02-10 13:00:43,767] vuegen: DEBUG - Processing subsection '14': 'Nestedness' - 5 component(s)
[2026-02-10 13:00:43,769] vuegen: INFO - Successfully generated content for DataFrame: 'Nestedness Random Subset'
[2026-02-10 13:00:43,770] vuegen: INFO - Successfully generated content for plot '35': 'All Samples'
[2026-02-10 13:00:43,771] vuegen: INFO - Successfully generated content for plot '36': 'Plant Samples'
[2026-02-10 13:00:43,772] vuegen: INFO - Successfully generated content for plot '37': 'Animal Samples'
[2026-02-10 13:00:43,774] vuegen: INFO - Successfully generated content for plot '38': 'Non Saline Samples'
[2026-02-10 13:00:43,774] vuegen: INFO - Generated content and imports for subsection: 'Nestedness'
[2026-02-10 13:00:43,775] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Metagenomics/Nestedness.py'
[2026-02-10 13:00:43,776] vuegen: DEBUG - Processing subsection '15': 'Shanon entropy analysis' - 1 component(s)
[2026-02-10 13:00:43,777] vuegen: INFO - Successfully generated content for plot '39': 'Specificity of sequences and higher taxonomic groups for environment'
[2026-02-10 13:00:43,778] vuegen: INFO - Generated content and imports for subsection: 'Shanon entropy analysis'
[2026-02-10 13:00:43,779] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Metagenomics/Shanon_entropy_analysis.py'
[2026-02-10 13:00:43,779] vuegen: DEBUG - Processing section '9': 'Network Analysis' - 1 subsection(s)
[2026-02-10 13:00:43,780] vuegen: DEBUG - Processing subsection '16': 'Phyla Association Networks' - 3 component(s)
[2026-02-10 13:00:43,782] vuegen: INFO - Successfully generated content for DataFrame: 'Phyla Counts Subset'
[2026-02-10 13:00:43,784] vuegen: INFO - Successfully read network from file: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/example_data/Earth_microbiome_vuegen_demo_notebook/3_Network_analysis/1_phyla_association_networks/2_phyla_correlation_network_with_0.5_threshold_edgelist.csv.
[2026-02-10 13:00:43,785] vuegen: DEBUG - Try to save network as PyVis network: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/streamlit_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:43,799] vuegen: INFO - PyVis network created and saved as: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/streamlit_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:43,801] vuegen: INFO - Successfully generated content for plot '40': 'Phyla Correlation Network With 0.5 Threshold Edgelist'
[2026-02-10 13:00:43,802] vuegen: INFO - Successfully generated content for plot '41': 'Phyla Correlation Network With 0.5 Threshold'
[2026-02-10 13:00:43,802] vuegen: INFO - Generated content and imports for subsection: 'Phyla Association Networks'
[2026-02-10 13:00:43,804] vuegen: INFO - Subsection file created: 'streamlit_report/sections/Network_Analysis/Phyla_Association_Networks.py'
[2026-02-10 13:00:43,805] vuegen: INFO - All the scripts to build the Streamlit app are available at streamlit_report/sections
[2026-02-10 13:00:43,805] vuegen: INFO - To run the Streamlit app, use the following command:
[2026-02-10 13:00:43,806] vuegen: INFO - streamlit run streamlit_report/sections/report_manager.py
All the scripts to build the Streamlit app are available at: streamlit_report/sections
To run the Streamlit app, use the following command:
streamlit run streamlit_report/sections/report_manager.py
run_streamlit = False
# run_streamlit = True # uncomment line to run the streamlit report
# Launch the Streamlit report depneding on the platform
if not IN_COLAB and run_streamlit:
!streamlit run streamlit_report/sections/report_manager.py
elif run_streamlit:
# see: https://discuss.streamlit.io/t/how-to-launch-streamlit-app-from-google-colab-notebook/42399
print(
"Password/Enpoint IP for localtunnel is:",
urllib.request.urlopen("https://ipv4.icanhazip.com")
.read()
.decode("utf8")
.strip("\n"),
)
# Run the Streamlit app in the background
!streamlit run streamlit_report/sections/report_manager.py --server.address=localhost &>/content/logs.txt &
# Expose the Streamlit app on port 8501
!npx localtunnel --port 8501 --subdomain vuegen-demo
else:
print("Streamlit report not executed, set run_streamlit to True to run the report")
Show code cell output
Hide code cell output
Streamlit report not executed, set run_streamlit to True to run the report
5.6. HTML and other report types with the extended configuration file#
# Test the changes by generarating the report from the modified YAML file
report_type = "html"
_ = report_generator.get_report(
config_path=config_path, report_type=report_type, logger=None
)
Show code cell output
Hide code cell output
[2026-02-10 13:00:43,822] vuegen: INFO - Path to log file: logs/2026-02-10_13-00-43_report.log
[2026-02-10 13:00:43,836] vuegen: INFO - Report 'Earth Microbiome Vuegen Demo Notebook' initialized with 3 sections.
[2026-02-10 13:00:43,837] vuegen: INFO - running in a normal Python process
[2026-02-10 13:00:43,838] vuegen: DEBUG - env_vars (QuartoReport): environ({'READTHEDOCS_VIRTUALENV_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable', 'READTHEDOCS_CANONICAL_URL': 'https://vuegen.readthedocs.io/en/stable/', 'HOSTNAME': 'build-31350056-project-1135329-vuegen', 'READTHEDOCS_GIT_CLONE_URL': 'https://github.com/Multiomics-Analytics-Group/vuegen.git', 'HOME': '/home/docs', 'NO_COLOR': '1', 'READTHEDOCS': 'True', 'READTHEDOCS_PRODUCTION_DOMAIN': 'app.readthedocs.org', 'READTHEDOCS_REPOSITORY_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable', 'READTHEDOCS_PROJECT': 'vuegen', 'READTHEDOCS_OUTPUT': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/_readthedocs/', 'PATH': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/bin:/home/docs/.asdf/shims:/home/docs/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'READTHEDOCS_VERSION_TYPE': 'tag', 'LANG': 'C.UTF-8', 'READTHEDOCS_LANGUAGE': 'en', 'DEBIAN_FRONTEND': 'noninteractive', 'READTHEDOCS_GIT_COMMIT_HASH': '5258cc795a2ea011d75b5c1e450ade2f7b59c8a5', 'READTHEDOCS_VERSION_NAME': 'stable', 'READTHEDOCS_VERSION': 'stable', 'PWD': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs', 'READTHEDOCS_GIT_IDENTIFIER': 'v0.6.0', 'DOCUTILSCONFIG': '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/docutils.conf', 'MPLBACKEND': 'Agg', 'JPY_PARENT_PID': '698', 'PYDEVD_USE_FRAME_EVAL': 'NO', 'TERM': 'xterm-color', 'CLICOLOR': '1', 'FORCE_COLOR': '1', 'CLICOLOR_FORCE': '1', 'PAGER': 'cat', 'GIT_PAGER': 'cat'})
[2026-02-10 13:00:43,839] vuegen: DEBUG - PATH: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/bin:/home/docs/.asdf/shims:/home/docs/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[2026-02-10 13:00:43,840] vuegen: DEBUG - sys.path: ['/home/docs/.asdf/installs/python/3.11.12/lib/python311.zip', '/home/docs/.asdf/installs/python/3.11.12/lib/python3.11', '/home/docs/.asdf/installs/python/3.11.12/lib/python3.11/lib-dynload', '', '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/envs/stable/lib/python3.11/site-packages']
[2026-02-10 13:00:43,841] vuegen: DEBUG - Generating 'html' report in directory: '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report'
[2026-02-10 13:00:43,841] vuegen: DEBUG - Output directory already existed: '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report'
[2026-02-10 13:00:43,842] vuegen: INFO - Output directory for static content already existed: 'quarto_report/static'
[2026-02-10 13:00:43,843] vuegen: INFO - Starting to generate sections for the report.
[2026-02-10 13:00:43,844] vuegen: DEBUG - Processing section: 'Exploratory Data Analysis' - 1 subsection(s)
[2026-02-10 13:00:43,845] vuegen: DEBUG - Processing subsection: 'Sample Exploration' - 5 component(s)
[2026-02-10 13:00:43,846] vuegen: DEBUG - component_imports: ['import pandas as pd', 'from itables import show, init_notebook_mode', 'init_notebook_mode(all_interactive=True)']
[2026-02-10 13:00:43,848] vuegen: INFO - Successfully generated content for DataFrame: 'Metadata Random Subset'
[2026-02-10 13:00:43,848] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,849] vuegen: INFO - Successfully generated content for plot: 'Animal Samples Map'
[2026-02-10 13:00:43,850] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:43,851] vuegen: INFO - Successfully generated content for plot: 'Plant Samples Map'
[2026-02-10 13:00:43,852] vuegen: DEBUG - component_imports: ['import altair as alt', 'import requests', 'import json']
[2026-02-10 13:00:43,853] vuegen: INFO - Successfully generated content for plot: 'Saline Samples Map'
[2026-02-10 13:00:43,854] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,854] vuegen: INFO - Successfully generated content for plot: 'Physicochemical properties of the EMP samples'
[2026-02-10 13:00:43,855] vuegen: INFO - Generated content and imports for subsection: 'Sample Exploration'
[2026-02-10 13:00:43,856] vuegen: DEBUG - Processing section: 'Metagenomics' - 4 subsection(s)
[2026-02-10 13:00:43,856] vuegen: DEBUG - Processing subsection: 'Alpha Diversity' - 2 component(s)
[2026-02-10 13:00:43,857] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,858] vuegen: INFO - Successfully generated content for plot: 'Alpha Diversity Host Associated Samples'
[2026-02-10 13:00:43,859] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:43,860] vuegen: INFO - Successfully generated content for plot: 'Alpha Diversity Free Living Samples'
[2026-02-10 13:00:43,861] vuegen: INFO - Generated content and imports for subsection: 'Alpha Diversity'
[2026-02-10 13:00:43,861] vuegen: DEBUG - Processing subsection: 'Average Copy Number' - 2 component(s)
[2026-02-10 13:00:43,862] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,863] vuegen: INFO - Successfully generated content for plot: 'Average Copy Number Emp Ontology Level2'
[2026-02-10 13:00:43,864] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:43,865] vuegen: INFO - Successfully generated content for plot: 'Average Copy Number Emp Ontology Level3'
[2026-02-10 13:00:43,866] vuegen: INFO - Generated content and imports for subsection: 'Average Copy Number'
[2026-02-10 13:00:43,866] vuegen: DEBUG - Processing subsection: 'Nestedness' - 5 component(s)
[2026-02-10 13:00:43,867] vuegen: DEBUG - component_imports: ['import pandas as pd', 'from itables import show, init_notebook_mode', 'init_notebook_mode(all_interactive=True)']
[2026-02-10 13:00:43,869] vuegen: INFO - Successfully generated content for DataFrame: 'Nestedness Random Subset'
[2026-02-10 13:00:43,869] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:43,871] vuegen: INFO - Successfully generated content for plot: 'All Samples'
[2026-02-10 13:00:43,871] vuegen: DEBUG - component_imports: ['import plotly.io as pio', 'import requests', 'import json']
[2026-02-10 13:00:43,873] vuegen: INFO - Successfully generated content for plot: 'Plant Samples'
[2026-02-10 13:00:43,873] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,874] vuegen: INFO - Successfully generated content for plot: 'Animal Samples'
[2026-02-10 13:00:43,875] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,876] vuegen: INFO - Successfully generated content for plot: 'Non Saline Samples'
[2026-02-10 13:00:43,876] vuegen: INFO - Generated content and imports for subsection: 'Nestedness'
[2026-02-10 13:00:43,877] vuegen: DEBUG - Processing subsection: 'Shanon entropy analysis' - 1 component(s)
[2026-02-10 13:00:43,877] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,879] vuegen: INFO - Successfully generated content for plot: 'Specificity of sequences and higher taxonomic groups for environment'
[2026-02-10 13:00:43,879] vuegen: INFO - Generated content and imports for subsection: 'Shanon entropy analysis'
[2026-02-10 13:00:43,880] vuegen: DEBUG - Processing section: 'Network Analysis' - 1 subsection(s)
[2026-02-10 13:00:43,880] vuegen: DEBUG - Processing subsection: 'Phyla Association Networks' - 3 component(s)
[2026-02-10 13:00:43,881] vuegen: DEBUG - component_imports: ['import pandas as pd', 'from itables import show, init_notebook_mode', 'init_notebook_mode(all_interactive=True)']
[2026-02-10 13:00:43,882] vuegen: INFO - Successfully generated content for DataFrame: 'Phyla Counts Subset'
[2026-02-10 13:00:43,884] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,886] vuegen: INFO - Successfully read network from file: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/example_data/Earth_microbiome_vuegen_demo_notebook/3_Network_analysis/1_phyla_association_networks/2_phyla_correlation_network_with_0.5_threshold_edgelist.csv.
[2026-02-10 13:00:43,887] vuegen: DEBUG - Try to save network as PyVis network: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:43,901] vuegen: INFO - PyVis network created and saved as: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/static/Phyla_Correlation_Network_With_0.5_Threshold_Edgelist.html.
[2026-02-10 13:00:43,903] vuegen: INFO - Successfully generated content for plot: 'Phyla Correlation Network With 0.5 Threshold Edgelist'
[2026-02-10 13:00:43,904] vuegen: DEBUG - component_imports: []
[2026-02-10 13:00:43,905] vuegen: INFO - Successfully generated content for plot: 'Phyla Correlation Network With 0.5 Threshold'
[2026-02-10 13:00:43,906] vuegen: INFO - Generated content and imports for subsection: 'Phyla Association Networks'
[2026-02-10 13:00:43,907] vuegen: INFO - Created qmd script to render the app: /home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/quarto_report.qmd
[2026-02-10 13:00:43,908] vuegen: INFO - Running 'Earth Microbiome Vuegen Demo Notebook' 'html' report with ['quarto', 'render', '/home/docs/checkouts/readthedocs.org/user_builds/vuegen/checkouts/stable/docs/quarto_report/quarto_report.qmd']
[2026-02-10 13:01:06,663] vuegen: INFO - 'Earth Microbiome Vuegen Demo Notebook' 'html' report rendered
Starting python3 kernel...
Done
Executing 'quarto_report.quarto_ipynb'
Cell 1/10: 'Imports'......................................
Done
Cell 2/10: 'Metadata Random Subset 10'....................Done
Cell 3/10: 'Plant Samples Map 43'.........................Done
Cell 4/10: 'Saline Samples Map 44'........................
Done
Cell 5/10: 'Alpha Diversity Free Living Samples 47'.......Done
Cell 6/10: 'Average Copy Number Emp Ontology Level3 49'...
Done
Cell 7/10: 'Nestedness Random Subset 11'..................Done
Cell 8/10: 'All Samples 50'...............................
Done
Cell 9/10: 'Plant Samples 51'.............................Done
Cell 10/10: 'Phyla Counts Subset 12'.......................Done
pandoc
to: html
output-file: quarto_report.html
standalone: true
self-contained: true
section-divs: true
html-math-method: mathjax
wrap: none
default-image-extension: png
toc: true
toc-depth: 3
variables: {}
metadata
document-css: false
link-citations: true
date-format: long
lang: en
title: Earth Microbiome Vuegen Demo Notebook
jupyter: python3
toc-location: left
page-layout: full
Output created: quarto_report.html