Taxoniq: Taxon Information Query - fast, offline querying of NCBI Taxonomy and related data

Taxoniq is a Python and command-line interface to the NCBI Taxonomy database and selected data sources that cross-reference it.

Taxoniq’s features include:

• Pre-computed indexes updated monthly from NCBI, WoL and cross-referenced databases

• Offline operation: all indexes are bundled with the package; no network calls are made when querying taxon information (separately, Taxoniq can fetch the nucleotide or protein sequences over the network given a taxon or sequence accession ID - see Retrieving sequences below)

• A CLI capable of JSON I/O, batch processing and streaming of inputs for ease of use and pipelining in shell scripts

• A stable, well-documented, type-hinted Python API (Python 3.6 and higher is supported)

• Comprehensive testing and continuous integration

• An intuitive interface with useful defaults

• Compactness, readability, and extensibility

The Taxoniq package bundles an indexed, compressed copy of the NCBI taxonomy database files, the NCBI RefSeq nucleotide and protein sequence accession IDs associated with each taxon, the WoL kingdom-wide phylogenomic distance database, and relevant information from other databases. Sequence accession IDs which appear in the NCBI RefSeq BLAST databases are indexed so that given a taxon ID, accession ID, or taxon name, you can quickly retrieve the taxon’s rank, lineage, description, citations, representative RefSeq IDs, LCA information, evolutionary distance, sequence (with a network call), and more, as described in the Cookbook section below. Full API documentation is available.

Installation

pip3 install taxoniq

Pre-built wheels are available for Python 3.5+ on Linux and MacOS. On MacOS 11 Big Sur, Pip 20.3+ is required to install pre-built wheels (you can check your version with pip3 --version and upgrade with pip3 install --upgrade pip).

Synopsis

>>> import taxoniq
>>> t = taxoniq.Taxon(9606)
>>> t.scientific_name
'Homo sapiens'
>>> t.common_name
'human'

>>> t.ranked_lineage
[taxoniq.Taxon(9606), taxoniq.Taxon(9605), taxoniq.Taxon(9604), taxoniq.Taxon(9443),
 taxoniq.Taxon(40674), taxoniq.Taxon(7711), taxoniq.Taxon(33208), taxoniq.Taxon(2759)]
>>> len(t.lineage)
32
>>> [(t.rank.name, t.scientific_name) for t in t.ranked_lineage]
[('species', 'Homo sapiens'), ('genus', 'Homo'), ('family', 'Hominidae'), ('order', 'Primates'),
 ('class', 'Mammalia'), ('phylum', 'Chordata'), ('kingdom', 'Metazoa'), ('superkingdom', 'Eukaryota')]
>>> [(c.rank.name, c.common_name) for c in t.child_nodes]
[('subspecies', 'Neandertal'), ('subspecies', 'Denisova hominin')]

>>> t.refseq_representative_genome_accessions[:10]
[taxoniq.Accession('NC_000001.11'), taxoniq.Accession('NC_000002.12'), taxoniq.Accession('NC_000003.12'),
 taxoniq.Accession('NC_000004.12'), taxoniq.Accession('NC_000005.10'), taxoniq.Accession('NC_000006.12'),
 taxoniq.Accession('NC_000007.14'), taxoniq.Accession('NC_000008.11'), taxoniq.Accession('NC_000009.12'),
 taxoniq.Accession('NC_000010.11')]

>>> t.url
'https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=9606'

# Wikidata provides structured links to many databases about taxa represented on Wikipedia
>>> t.wikidata_url
'https://www.wikidata.org/wiki/Q15978631'

>>> t2 = taxoniq.Taxon(scientific_name="Bacillus anthracis")
>>> t2.description
'<p class="mw-empty-elt"> </p> <p class="mw-empty-elt"> </p> <p><i><b>Bacillus anthracis</b></i>
 is the agent of anthrax—a common disease of livestock and, occasionally, of humans—and the only
 obligate pathogen within the genus <i>Bacillus</i>. This disease can be classified as a zoonosis,
 causing infected animals to transmit the disease to humans. <i>B. anthracis</i> is a Gram-positive,
 endospore-forming, rod-shaped bacterium, with a width of 1.0–1.2 µm and a length of 3–5&#160;µm.
 It can be grown in an ordinary nutrient medium under aerobic or anaerobic conditions.</p>
 <p>It is one of few bacteria known to synthesize a protein capsule (poly-D-gamma-glutamic acid).
 Like <i>Bordetella pertussis</i>, it forms a calmodulin-dependent adenylate cyclase exotoxin known
 as anthrax edema factor, along with anthrax lethal factor. It bears close genotypic and phenotypic
 resemblance to <i>Bacillus cereus</i> and <i>Bacillus thuringiensis</i>. All three species share
 cellular dimensions and morphology</p>...'

>>> t3 = taxoniq.Taxon(accession_id="NC_000913.3")
>>> t3.scientific_name
'Escherichia coli str. K-12 substr. MG1655"'
>>> t3.parent.parent.common_name
'E. coli'
>>> t3.refseq_representative_genome_accessions[0].length
4641652

# The get_from_s3() method is the only command that will trigger a network call.
>>> seq = t3.refseq_representative_genome_accessions[0].get_from_s3().read()
>>> len(seq)
4641652
>>> seq[:64]
b'AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGAT'

Retrieving sequences

Mirrors of the NCBI BLAST databases are maintained on AWS S3 (s3://ncbi-blast-databases) and Google Storage (gs://blast-db). This is a key resource, since S3 and GS have superior bandwidth and throughput compared to the NCBI FTP server, so range requests can be used to retrieve individual sequences from the database files without downloading and keeping a copy of the whole database.

The Taxoniq PyPI distribution (the package you install using pip3 install taxoniq) indexes sequence accession IDs for the following NCBI BLAST databases:

• Refseq viruses representative genomes (ref_viruses_rep_genomes) (nucleotide)

• Refseq prokaryote representative genomes (contains refseq assembly) (ref_prok_rep_genomes) (nucleotide)

• RefSeq Eukaryotic Representative Genome Database (ref_euk_rep_genomes) (nucleotide)

• Betacoronavirus (nucleotide)

Given an accession ID, Taxoniq can issue a single HTTP request and return a file-like object streaming the nucleotide sequence from the S3 or GS mirror as follows:

with taxoniq.Accession("NC_000913.3").get_from_s3() as fh:
    fh.read()

For brevity, you can use urllib3.response.HTTPResponse.stream instead of read(...) to avoid holding the entire sequence in memory:

with taxoniq.Accession("NC_000913.3").get_from_s3() as fh:
    for chunk in fh.stream():
        sys.stdout.buffer.write(chunk)

To retrieve many sequences quickly, you may want to use a threadpool to open multiple network connections at once:

from concurrent.futures import ThreadPoolExecutor
def fetch_seq(accession):
    seq = accession.get_from_s3().read()
    return (accession, seq)

taxon = taxoniq.Taxon(scientific_name="Apis mellifera")
for accession, seq in ThreadPoolExecutor().map(fetch_seq, taxon.refseq_representative_genome_accessions):
    print(accession, len(seq))

This operation is also available in the CLI, as described below.

Command-line interface

pip3 install taxoniq installs a command-line utility, taxoniq, which can be used to perform many of the same functions provided by the Python API:

>taxoniq child-nodes --taxon-id 2 --output-format '{tax_id}: {scientific_name}'
[
    "1224: Proteobacteria",
    "2323: Bacteria incertae sedis",
    "32066: Fusobacteria",
    "40117: Nitrospirae",
    "48479: environmental samples",
    "49928: unclassified Bacteria",
    "57723: Acidobacteria",
    "68297: Dictyoglomi",
    "74152: Elusimicrobia",
    "200783: Aquificae",
    "200918: Thermotogae",
    "200930: Deferribacteres",
    "200938: Chrysiogenetes",
    "200940: Thermodesulfobacteria",
    "203691: Spirochaetes",
    "508458: Synergistetes",
    "1783257: PVC group",
    "1783270: FCB group",
    "1783272: Terrabacteria group",
    "1802340: Nitrospinae/Tectomicrobia group",
    "1930617: Calditrichaeota",
    "2138240: Coprothermobacterota",
    "2498710: Caldiserica/Cryosericota group",
    "2698788: Candidatus Krumholzibacteriota",
    "2716431: Coleospermum",
    "2780997: Vogosella"
]

See taxoniq --help for full details.

Retrieving sequences using the CLI

To retrieve an individual sequence in FASTA format given an accession ID, use taxoniq get-from-s3 --accession-id ACCESSION_ID.

To retrieve multiple sequences in FASTA format, use --accession-id - and pass the IDs on standard input, one per line: taxoniq refseq-representative-genome-accessions --scientific-name="Apis mellifera" | jq -r .[] | taxoniq get-from-s3 --accession-id -.

Using the nr/nt databases

Because of their size, taxoniq wheels with indexes of the NT (GenBank Non-redundant nucleotide) BLAST database are distributed on GitHub instead of PyPI. After running pip3 install taxoniq, you can install the NT indexes as follows:

• Navigate to https://github.com/chanzuckerberg/taxoniq/releases/latest

• In the “Assets” section, for each link that starts with “ncbi_genbank” and ends with “.whl”:

  • Right-click on the asset link, and click “Copy link address”

  • Run pip3 install --upgrade <PASTED LINK ADDRESS>

The NT index packages also contain indexes for the RefSeq representative genomes and Betacoronavirus accessions (meaning they are are superset of the PyPI packages).

Streaming CLI I/O

The taxoniq command-line interface can take streaming input from stdin and produce streaming output on stdout. This allows the amortization of startup and index load time and efficient operation as part of shell pipelines.

The following example shows the pipelined operation of fastp, kraken2, and taxoniq to annotate hits found in a Betacoronavirus sample:

in progress

Cookbook

In progress

Links

• Project home page (GitHub)

• Documentation

• Package distribution (PyPI)

• Change log

• ETE Toolkit and the ETE Tutorial - a general purpose phylogenetic tree toolkit with a rich Python interface

License

Taxoniq software is licensed under the terms of the MIT License.

Distributions of this package contain data from NCBI Taxonomy, NCBI GenBank, and NCBI RefSeq (Bethesda (MD): National Library of Medicine (US), National Center for Biotechnology Information). These data are released into the public domain under the NCBI Public Domain Notice.

Distributions of this package contain text excerpts from Wikipedia licensed under the terms of the CC-BY-SA License.

Bugs

Please report bugs, issues, feature requests, etc. on GitHub.

API documentation

class taxoniq.Accession(accession_id: str)

An object representing an NCBI GenBank nucleotide or protein sequence accession ID. This is used by Taxoniq to represent sequences associated with taxons; use Taxon as the starting point.

property blast_db

The BLAST database in which this sequence accession ID was indexed.

property blast_db_volume

The numeric BLAST database volume ID in which this sequence accession was indexed.

property db_offset

The byte offset in the BLAST database volume at which this sequence starts.

get_from_gs()

Returns a file-like object streaming the nucleotide sequence for this accession from the Google Storage NCBI BLAST database mirror (https://registry.opendata.aws/ncbi-blast-databases/), if available.

get_from_s3()

Returns a file-like object streaming the nucleotide sequence for this accession from the AWS S3 NCBI BLAST database mirror (https://registry.opendata.aws/ncbi-blast-databases/), if available.

property length

The length of the sequence (number of nucleotides or amino acids).

property tax_id

The taxon ID associated with this sequence accession ID.

url()

Returns the HTTPS URL for the NCBI GenBank web page representing this sequence accession ID.

class taxoniq.BLASTDatabase(value)

An enumeration.

exception taxoniq.NoValue

class taxoniq.Rank(value)

An enumeration.

class taxoniq.Taxon(tax_id: Optional[int] = None, accession_id: Optional[str] = None, scientific_name: Optional[str] = None)

An object representing an NCBI Taxonomy taxon, identified by its taxon ID. The object can be instantiated by uniquely identifying a taxon using the numeric taxon ID, an alphanumeric accession ID of a sequence associated with the taxon ID, or the scientific name of the taxon.

property best_available_description

Introductory paragraph from English Wikipedia for this taxon or the first parent taxon where a description is available.

property best_refseq_taxon

Best related taxon with refseq representative genome sequence available. For viruses, this is the RefSeq “genome neighbor” as defined in https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4383986/ and retrieved from https://ftp.ncbi.nlm.nih.gov/genomes/Viruses/Viruses_RefSeq_and_neighbors_genome_data.tab. For other domains, this is the RefSeq representative genome for the taxon’s species, if available, as seen in the species_taxid column of https://ftp.ncbi.nlm.nih.gov/genomes/refseq/assembly_summary_refseq.txt.

The accessions for the genome can be accessed as follows:

Taxon(123).best_refseq_taxon.refseq_representative_genome_accessions

property child_nodes

Returns a list of taxon objects that list this taxon as their parent.

closest_taxon_with_refseq_genome()

Returns a taxon closest by phylogenetic distance as computed by WoL and with a refseq genome associated

property common_name

Common name of the taxon. In taxoniq, this is defined as the NCBI taxonomy blast name if available, or the genbank common name if available, or the first listed common name. See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3245000/ for definitions of these fields.

property description

Introductory paragraph for this taxon from English Wikipedia, if available.

classmethod distance(taxa)

Phylogenetic distance between taxa as computed by WoL

property host

A text description of a symbiont host or hosts for this taxon’s organism, if any.

classmethod lca(taxa)

Given a list of Taxon objects, returns the last common ancestor taxon.

property lineage

Lineage for this taxon (the list of parent nodes from the taxon to the root of the taxonomic tree).

property parent

The parent taxon for this taxon.

For the root of the tree, parent is None.

property rank

Rank of the taxon. See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7408187/#sec9title for more details.

property ranked_child_nodes

List of child nodes in the next main taxonomic rank (species, genus, family, order, class, phylum, kingdom, superkingdom).

property ranked_lineage

Lineage of main taxonomic ranks (species, genus, family, order, class, phylum, kingdom, superkingdom).

property refseq_genome_accessions

A list of Accession objects for sequences in the most recent RefSeq genome assembly for this taxon, if available.

property refseq_representative_genome_accessions

A list of Accession objects for sequences in the RefSeq representative genome assembly for this taxon, if available.

property scientific_name

The unique scientific name of the taxon.

property url

Returns the HTTPS URL for the NCBI Taxonomy web page representing this taxon.

property wikidata_id

Wikidata ID for this taxon.

property wikidata_url

URL of the Wikidata web page representing this taxon.

exception taxoniq.TaxoniqException

Change log

• Release Notes
• Changes for v0.5.2 (2021-04-14)
• Changes for v0.5.1 (2021-04-14)
• Changes for v0.5.0 (2021-04-13)
• Changes for v0.4.0 (2021-04-02)
• Changes for v0.3.4 (2021-04-01)
• Changes for v0.3.3 (2021-04-01)
• Changes for v0.3.1 (2021-04-01)
• Changes for v0.3.0 (2021-03-27)
• Changes for v0.2.0 (2021-03-26)
• Changes for v0.1.7 (2021-03-25)
• Changes for v0.1.6 (2021-03-25)
• Changes for v0.1.5 (2021-03-25)
• Changes for v0.1.4 (2021-03-25)