Next Generation Sequencing has introduced a massive need for working with integer interval data which correspond to actual chromosomal regions, depicted in linear representations. As a result, previously under-developed algorithms for working with such data have tremendously evolved. Maybe the most common application where genomic intervals are used is overlapping a set of query intervals with a set of reference intervals. One typical example is counting the reads produced e.g. from an RNA-Seq experiment and assigning them to genes of interest through overlapping their mapped coordinates with those of the genes over a reference genome. As a result, collections of such reference genomic regions for several reference organisms are essential for the quick interrogation of the latter.
The generation of genomic coordinate systems are nowadays mainstream. Typical ways of reference genomic region representations are:
Bioconductor offers great infrastructures for fast genomic interval calculations which are now very mature, high-level and cover most needs. It also offers many comprehensive and centrally maintained genomic interval annotation packages as well as tools to quickly create custom annotation packages, such as AnnotationForge. These packages, are primarily designed to capture genomic structures (genes, transcripts, exons etc.) accurately and place them in a genomic interval content suitable for fast calculations. While this is more than sufficient for many users and work out-of-the-box, especially for less experienced R users, they may miss certain characteristics which may be useful also for many users. Such additional elements are often required by tools that report e.g. transcript biotypes (such as those in Ensembl) and do not gather mappings between elements of the same annotation (e.g. gene, transcript, exon ids) in one place in a more straightforward manner. More specifically, some elements which are not directly achievable with standard Bioconductor annotation packages include:
SiTaDelA (Simple Tab Delimited Annotations), through efficient
and extensive usage of Bioconductor facilites offers these additional
functionalities along with certain levels of automation. More specifically, the
sitadela package offers:
The sitadela annotation database building is extremely simple. The user
defines a list of desired annotations (organisms, sources, versions) and
supplies them to the addAnnotation function which in turn creates a new or
updates a current database. A custom, non-directly supported organism annotation
can be imported through the addCustomAnnotation function and annotations not
needed anymore can be removed with the removeAnnotation function. Finally, as
the building can require some time, especially if many organisms and sources are
required for a local database, we maintain pre-built databases which are built
periodically (e.g. upon a new Ensembl release).
The following organisms (essentially genome versions) are supported for automatic database builds:
Please note that if genomic annotations from UCSC, RefSeq or NCBI are required,
the following BSgenome packages are required (depending on the organisms to
be installed) in order to calculate GC content for gene annotations. Also note
that there is no BSgenome package for some of the sitadela supported
organisms and therefore GC contents will not be available anyway.
Is is therefore advised to install these BSgenome packages in advance.
To install the sitadela package, one should start R and enter:
if(!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("sitadela")By default, the database file will be written in the
system.file(package="sitadela") directory. You can specify another prefered
destination for it using the db argument in the function call, but if you do
that, you will have to supply an argument pointing to the SQLite database file
you created to every sitadela package function call you perform, or any other
function that uses sitadela annotations, otherwise, the annotation will be
downloaded and formatted on-the-fly instead of using the local database. Upon
loading sitadela, an option is added to the R environment pointing to the
default sitadela annotation database. If you wish to change that location and
do not wish to supply the database to other function calls, you can change the
default location of the annotation to your preferred location with the
setDbPath function in the beginning of your script/function that uses the
annotation database.
In this vignette, we will build a minimal database comprising only the mouse
mm9 genome version from Ensembl. The database will be built in a temporary
directory inside session tempdir().
Important note: As the annotation build function makes use of Kent utilities for creating 3’UTR annotations from RefSeq and UCSC, the latter cannot be built in Windows. Therefore it is advised to either build the annotation database in a Linux system or use our pre-built databases.
library(sitadela)
buildDir <- file.path(tempdir(),"test_anndb")
dir.create(buildDir)
# The location of the custom database
myDb <- file.path(buildDir,"testann.sqlite")
# Since we are using Ensembl, we can also ask for a version
organisms <- list(mm9=67)
sources <- ifelse(.Platform$OS.type=="unix",c("ensembl","refseq"),"ensembl")
# If the example is not running in a multicore system, rc is ignored
addAnnotation(organisms,sources,forceDownload=FALSE,db=myDb,rc=0.5)##
## ********************************************************## This is sitadela 1.0.0 genomic region annotation builder## ********************************************************## sitadela database found at /tmp/RtmppfVC9v/test_anndb directory##
## ========================================================## 2021-05-19 18:52:20 - Try 1## ========================================================## Opening sitadela SQLite database /tmp/RtmppfVC9v/test_anndb/testann.sqlite## Retrieving genome information for mm9 from ensembl## Loading required package: BiocGenerics## Loading required package: parallel##
## Attaching package: 'BiocGenerics'## The following objects are masked from 'package:parallel':
##
## clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
## clusterExport, clusterMap, parApply, parCapply, parLapply,
## parLapplyLB, parRapply, parSapply, parSapplyLB## The following objects are masked from 'package:stats':
##
## IQR, mad, sd, var, xtabs## The following objects are masked from 'package:base':
##
## Filter, Find, Map, Position, Reduce, anyDuplicated, append,
## as.data.frame, basename, cbind, colnames, dirname, do.call,
## duplicated, eval, evalq, get, grep, grepl, intersect, is.unsorted,
## lapply, mapply, match, mget, order, paste, pmax, pmax.int, pmin,
## pmin.int, rank, rbind, rownames, sapply, setdiff, sort, table,
## tapply, union, unique, unsplit, which.max, which.min## Loading required package: S4Vectors## Loading required package: stats4##
## Attaching package: 'S4Vectors'## The following objects are masked from 'package:base':
##
## I, expand.grid, unname## Retrieving gene annotation for mm9 from ensembl version 67## Using Ensembl host https://may2012.archive.ensembl.org## Retrieving transcript annotation for mm9 from ensembl version 67## Using Ensembl host https://may2012.archive.ensembl.org## Merging transcripts for mm9 from ensembl version 67## Retrieving 3' UTR annotation for mm9 from ensembl version 67## Using Ensembl host https://may2012.archive.ensembl.org## Merging gene 3' UTRs for mm9 from ensembl version 67## Merging transcript 3' UTRs for mm9 from ensembl version 67## Retrieving exon annotation for mm9 from ensembl version 67## Using Ensembl host https://may2012.archive.ensembl.org## Retrieving extended exon annotation for mm9 from ensembl version 67## Using Ensembl host https://may2012.archive.ensembl.org## Merging exons for mm9 from ensembl version 67
## Merging exons for mm9 from ensembl version 67##
## -------------------------------------------------------## Building process complete!## -------------------------------------------------------## Alternatively
# setDbPath(myDb)
# addAnnotation(organisms,sources,forceDownload=FALSE,rc=0.5)Now, that a small database is in place, let’s retrieve some data. Remember that
since the built database is not in the default location, we need to pass the
database file in each data retrieval function. The annotation is retrieved as
a GRanges object by default.
# Load standard annotation based on gene body coordinates
genes <- loadAnnotation(genome="mm9",refdb="ensembl",type="gene",db=myDb)
genes## GRanges object with 37583 ranges and 4 metadata columns:
## seqnames ranges strand | gene_id
## <Rle> <IRanges> <Rle> | <character>
## ENSMUSG00000090025 chr1 3044314-3044814 + | ENSMUSG00000090025
## ENSMUSG00000064842 chr1 3092097-3092206 + | ENSMUSG00000064842
## ENSMUSG00000051951 chr1 3195982-3661579 - | ENSMUSG00000051951
## ENSMUSG00000089699 chr1 3456668-3503634 + | ENSMUSG00000089699
## ENSMUSG00000088390 chr1 3668961-3669024 - | ENSMUSG00000088390
## ... ... ... ... . ...
## ENSMUSG00000052831 chrY 2086590-2097768 + | ENSMUSG00000052831
## ENSMUSG00000069031 chrY 2118049-2129045 + | ENSMUSG00000069031
## ENSMUSG00000071960 chrY 2156899-2168120 + | ENSMUSG00000071960
## ENSMUSG00000091987 chrY 2390390-2398856 + | ENSMUSG00000091987
## ENSMUSG00000090600 chrY 2550262-2552957 + | ENSMUSG00000090600
## gc_content gene_name biotype
## <numeric> <character> <character>
## ENSMUSG00000090025 48.30 Gm16088 pseudogene
## ENSMUSG00000064842 36.36 U6 snRNA
## ENSMUSG00000051951 38.51 Xkr4 protein_coding
## ENSMUSG00000089699 39.27 Gm1992 antisense
## ENSMUSG00000088390 34.38 U7 snRNA
## ... ... ... ...
## ENSMUSG00000052831 36.60 Rbmy1a1 protein_coding
## ENSMUSG00000069031 37.06 Gm10256 protein_coding
## ENSMUSG00000071960 37.12 Gm10352 protein_coding
## ENSMUSG00000091987 34.84 Gm3376 protein_coding
## ENSMUSG00000090600 44.40 Gm3395 protein_coding
## -------
## seqinfo: 21 sequences from mm9 genome# Load standard annotation based on 3' UTR coordinates
utrs <- loadAnnotation(genome="mm9",refdb="ensembl",type="utr",db=myDb)
utrs## GRanges object with 161406 ranges and 4 metadata columns:
## seqnames ranges strand | transcript_id
## <Rle> <IRanges> <Rle> | <character>
## ENSMUST00000160944 chr1 3044815-3045092 + | ENSMUST00000160944
## ENSMUST00000082908 chr1 3092207-3092484 + | ENSMUST00000082908
## ENSMUST00000162897 chr1 3195704-3195981 - | ENSMUST00000162897
## ENSMUST00000159265 chr1 3196326-3196603 - | ENSMUST00000159265
## ENSMUST00000070533 chr1 3204285-3204562 - | ENSMUST00000070533
## ... ... ... ... . ...
## ENSMUST00000078383 chrY 2167760-2168120 + | ENSMUST00000078383
## ENSMUST00000078383 chrY 2168121-2168398 + | ENSMUST00000078383
## ENSMUST00000166474 chrY 2398496-2398856 + | ENSMUST00000166474
## ENSMUST00000166474 chrY 2398857-2399134 + | ENSMUST00000166474
## ENSMUST00000172100 chrY 2552084-2552957 + | ENSMUST00000172100
## gene_id gene_name biotype
## <character> <character> <character>
## ENSMUST00000160944 ENSMUSG00000090025 Gm16088 pseudogene
## ENSMUST00000082908 ENSMUSG00000064842 U6 snRNA
## ENSMUST00000162897 ENSMUSG00000051951 Xkr4 protein_coding
## ENSMUST00000159265 ENSMUSG00000051951 Xkr4 protein_coding
## ENSMUST00000070533 ENSMUSG00000051951 Xkr4 protein_coding
## ... ... ... ...
## ENSMUST00000078383 ENSMUSG00000071960 Gm10352 protein_coding
## ENSMUST00000078383 ENSMUSG00000071960 Gm10352 protein_coding
## ENSMUST00000166474 ENSMUSG00000091987 Gm3376 protein_coding
## ENSMUST00000166474 ENSMUSG00000091987 Gm3376 protein_coding
## ENSMUST00000172100 ENSMUSG00000090600 Gm3395 protein_coding
## -------
## seqinfo: 21 sequences from mm9 genome# Load summarized exon annotation based used with RNA-Seq analysis
sumEx <- loadAnnotation(genome="mm9",refdb="ensembl",type="exon",
summarized=TRUE,db=myDb)
sumEx## GRanges object with 247808 ranges and 4 metadata columns:
## seqnames ranges strand |
## <Rle> <IRanges> <Rle> |
## ENSMUSG00000090025_MEX_1 chr1 3044314-3044814 + |
## ENSMUSG00000064842_MEX_1 chr1 3092097-3092206 + |
## ENSMUSG00000051951_MEX_4 chr1 3195982-3197398 - |
## ENSMUSG00000051951_MEX_3 chr1 3203520-3207049 - |
## ENSMUSG00000051951_MEX_2 chr1 3411783-3411982 - |
## ... ... ... ... .
## ENSMUSG00000091987_MEX_5 chrY 2394921-2395029 + |
## ENSMUSG00000091987_MEX_6 chrY 2396398-2396478 + |
## ENSMUSG00000091987_MEX_7 chrY 2397909-2397997 + |
## ENSMUSG00000091987_MEX_8 chrY 2398179-2398856 + |
## ENSMUSG00000090600_MEX_1 chrY 2550262-2552957 + |
## exon_id gene_id
## <character> <character>
## ENSMUSG00000090025_MEX_1 ENSMUSG00000090025_M.. ENSMUSG00000090025
## ENSMUSG00000064842_MEX_1 ENSMUSG00000064842_M.. ENSMUSG00000064842
## ENSMUSG00000051951_MEX_4 ENSMUSG00000051951_M.. ENSMUSG00000051951
## ENSMUSG00000051951_MEX_3 ENSMUSG00000051951_M.. ENSMUSG00000051951
## ENSMUSG00000051951_MEX_2 ENSMUSG00000051951_M.. ENSMUSG00000051951
## ... ... ...
## ENSMUSG00000091987_MEX_5 ENSMUSG00000091987_M.. ENSMUSG00000091987
## ENSMUSG00000091987_MEX_6 ENSMUSG00000091987_M.. ENSMUSG00000091987
## ENSMUSG00000091987_MEX_7 ENSMUSG00000091987_M.. ENSMUSG00000091987
## ENSMUSG00000091987_MEX_8 ENSMUSG00000091987_M.. ENSMUSG00000091987
## ENSMUSG00000090600_MEX_1 ENSMUSG00000090600_M.. ENSMUSG00000090600
## gene_name biotype
## <character> <character>
## ENSMUSG00000090025_MEX_1 Gm16088 pseudogene
## ENSMUSG00000064842_MEX_1 U6 snRNA
## ENSMUSG00000051951_MEX_4 Xkr4 protein_coding
## ENSMUSG00000051951_MEX_3 Xkr4 protein_coding
## ENSMUSG00000051951_MEX_2 Xkr4 protein_coding
## ... ... ...
## ENSMUSG00000091987_MEX_5 Gm3376 protein_coding
## ENSMUSG00000091987_MEX_6 Gm3376 protein_coding
## ENSMUSG00000091987_MEX_7 Gm3376 protein_coding
## ENSMUSG00000091987_MEX_8 Gm3376 protein_coding
## ENSMUSG00000090600_MEX_1 Gm3395 protein_coding
## -------
## seqinfo: 21 sequences from mm9 genome# Load standard annotation based on gene body coordinates from RefSeq
if (.Platform$OS.type=="unix") {
refGenes <- loadAnnotation(genome="mm9",refdb="refseq",type="gene",
db=myDb)
refGenes
}## Getting latest annotation on the fly for mm9 from refseq## Retrieving genome information for mm9 from refseq## Retrieving latest gene annotation for mm9 from refseq## Loading required namespace: RMySQL## Converting annotation to GenomicRanges object...## Loading required namespace: BSgenome##
## Attaching package: 'Biostrings'## The following object is masked from 'package:base':
##
## strsplit## Getting DNA sequences...## Getting GC content...## GRanges object with 20935 ranges and 4 metadata columns:
## seqnames ranges strand | gene_id gc_content
## <Rle> <IRanges> <Rle> | <character> <numeric>
## NM_001011874 chr1 3204562-3661579 - | NM_001011874 38.56
## NM_011283 chr1 4333587-4350395 - | NM_011283 38.73
## NM_011441 chr1 4481008-4487435 - | NM_011441 49.75
## NM_001177658 chr1 4763278-4775807 - | NM_001177658 42.59
## NM_008866 chr1 4797903-4836816 + | NM_008866 40.99
## ... ... ... ... . ... ...
## NM_148943 chrY 635403-796225 - | NM_148943 36.39
## NM_009571 chrY 1362122-1426357 - | NM_009571 36.58
## NM_001177569 chrY 1855008-1855344 + | NM_001177569 49.85
## NM_011564 chrY 1918380-1919568 - | NM_011564 55.17
## NM_001166384 chrY 2156898-2168120 + | NM_001166384 37.13
## gene_name biotype
## <character> <character>
## NM_001011874 Xkr4 protein_coding
## NM_011283 Rp1 protein_coding
## NM_011441 Sox17 protein_coding
## NM_001177658 Mrpl15 protein_coding
## NM_008866 Lypla1 protein_coding
## ... ... ...
## NM_148943 Usp9y protein_coding
## NM_009571 Zfy2 protein_coding
## NM_001177569 H2al2c protein_coding
## NM_011564 Sry protein_coding
## NM_001166384 Rbmy protein_coding
## -------
## seqinfo: 21 sequences (21 circular) from mm9 genomeOr as a data frame if you prefer using asdf=TRUE. The data frame however does
not contain metadata like Seqinfo to be used for any susequent validations:
# Load standard annotation based on gene body coordinates
genes <- loadAnnotation(genome="mm9",refdb="ensembl",type="gene",db=myDb,
asdf=TRUE)
head(genes)## chromosome start end gene_id gc_content strand gene_name
## 1 chr1 3044314 3044814 ENSMUSG00000090025 48.30 + Gm16088
## 2 chr1 3092097 3092206 ENSMUSG00000064842 36.36 + U6
## 3 chr1 3195982 3661579 ENSMUSG00000051951 38.51 - Xkr4
## 4 chr1 3456668 3503634 ENSMUSG00000089699 39.27 + Gm1992
## 5 chr1 3668961 3669024 ENSMUSG00000088390 34.38 - U7
## 6 chr1 3773818 3773879 ENSMUSG00000089420 38.71 - U7
## biotype
## 1 pseudogene
## 2 snRNA
## 3 protein_coding
## 4 antisense
## 5 snRNA
## 6 snRNAApart from the supported organisms and databases, you can add a custom annotation. Such an annotation can be:
This can be achieved through the usage of
GTF/GFF files, along
with some simple metadata that you have to provide for proper import to the
annotation database. This can be achieved through the usage of the
addCustomAnnotation function. Details on required metadata can be found
in the function’s help page.
Important note: Please note that importing a custom genome annotation
directly from UCSC (UCSC SQL database dumps) is not supported in Windows as the
process involves using the genePredToGtf which is not available for Windows.
Let’s try a couple of examples. The first one uses example GTF files shipped with the package. These are sample chromosomes from:
Below, we test custom building with reference sequence HE567025 of Atlantic cod:
gtf <- system.file(package="sitadela","extdata",
"gadMor1_HE567025.gtf.gz")
chrom <- system.file(package="sitadela","extdata",
"gadMor1_HE567025.txt.gz")
chromInfo <- read.delim(chrom,header=FALSE,row.names=1)
names(chromInfo) <- "length"
metadata <- list(
organism="gadMor1_HE567025",
source="sitadela_package",
chromInfo=chromInfo
)
tmpdb <- tempfile()
addCustomAnnotation(gtfFile=gtf,metadata=metadata,db=tmpdb)## Opening sitadela SQLite database /tmp/RtmppfVC9v/file10d4bb71609eab## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/gadMor1_HE567025.gtf.gz as GTF to make id map## Making id map## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/gadMor1_HE567025.gtf.gz as TxDb## Import genomic features from the file as a GRanges object ... OK
## Prepare the 'metadata' data frame ... OK
## Make the TxDb object ... OK
## Retrieving gene annotation for gadmor1_he567025 from sitadela_package version 20210519 from /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/gadMor1_HE567025.gtf.gz
## Retrieving transcript annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized transcript annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving 3' UTR annotation for gadmor1_he567025 from sitadela_package version 20210519
## 3' UTR annotation for gadmor1_he567025 from sitadela_package version 20210519 is not available in the provided GTF file.
## Retrieving summarized 3' UTR annotation per gene for gadmor1_he567025 from sitadela_package version 20210519
## 3' UTR annotation for gadmor1_he567025 from sitadela_package version 20210519 is not available in the provided GTF file.
## Retrieving summarized 3' UTR annotation per transcript for gadmor1_he567025 from sitadela_package version 20210519
## 3' UTR annotation for gadmor1_he567025 from sitadela_package version 20210519 is not available in the provided GTF file.
## Retrieving exon annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized exon annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving extended exon annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized transcript exon annotation for gadmor1_he567025 from sitadela_package version 20210519# Try to retrieve some data
g <- loadAnnotation(genome="gadMor1_HE567025",refdb="sitadela_package",
type="gene",db=tmpdb)
g## GRanges object with 48 ranges and 4 metadata columns:
## seqnames ranges strand | gene_id gc_content gene_name
## <Rle> <IRanges> <Rle> | <character> <numeric> <character>
## g8912 HE567025 66-6023 + | g8912 50 g8912
## g8913 HE567025 17576-54518 - | g8913 50 g8913
## g8914 HE567025 74456-75028 - | g8914 50 g8914
## g8915 HE567025 98451-108568 - | g8915 50 g8915
## g8916 HE567025 129805-168324 + | g8916 50 g8916
## ... ... ... ... . ... ... ...
## g8955 HE567025 960225-962523 + | g8955 50 g8955
## g8956 HE567025 969370-988129 - | g8956 50 g8956
## g8957 HE567025 989587-1008879 - | g8957 50 g8957
## g8958 HE567025 1018881-1041482 - | g8958 50 g8958
## g8959 HE567025 1044660-1068026 + | g8959 50 g8959
## biotype
## <character>
## g8912 gene
## g8913 gene
## g8914 gene
## g8915 gene
## g8916 gene
## ... ...
## g8955 gene
## g8956 gene
## g8957 gene
## g8958 gene
## g8959 gene
## -------
## seqinfo: 1 sequence from gadmor1_he567025 genome# Delete the temporary database
unlink(tmpdb)The next one is part of a custom annotation for the Ebola virus from UCSC:
gtf <- system.file(package="sitadela","extdata",
"eboVir3_KM034562v1.gtf.gz")
chrom <- system.file(package="sitadela","extdata",
"eboVir3_KM034562v1.txt.gz")
chromInfo <- read.delim(chrom,header=FALSE,row.names=1)
names(chromInfo) <- "length"
metadata <- list(
organism="gadMor1_HE567025",
source="sitadela_package",
chromInfo=chromInfo
)
tmpdb <- tempfile()
addCustomAnnotation(gtfFile=gtf,metadata=metadata,db=tmpdb)## Opening sitadela SQLite database /tmp/RtmppfVC9v/file10d4bb2ad2f98d## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/eboVir3_KM034562v1.gtf.gz as GTF to make id map## Making id map## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/eboVir3_KM034562v1.gtf.gz as TxDb## Import genomic features from the file as a GRanges object ... OK
## Prepare the 'metadata' data frame ... OK
## Make the TxDb object ... OK
## Retrieving gene annotation for gadmor1_he567025 from sitadela_package version 20210519 from /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/eboVir3_KM034562v1.gtf.gz
## Retrieving transcript annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized transcript annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving 3' UTR annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized 3' UTR annotation per gene for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized 3' UTR annotation per transcript for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving exon annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized exon annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving extended exon annotation for gadmor1_he567025 from sitadela_package version 20210519
## Retrieving summarized transcript exon annotation for gadmor1_he567025 from sitadela_package version 20210519# Try to retrieve some data
g <- loadAnnotation(genome="gadMor1_HE567025",refdb="sitadela_package",
type="gene",db=tmpdb)
g## GRanges object with 9 ranges and 4 metadata columns:
## seqnames ranges strand | gene_id gc_content gene_name
## <Rle> <IRanges> <Rle> | <character> <numeric> <character>
## NP KM034562v1 56-3026 + | NP 50 NP
## VP35 KM034562v1 3032-4407 + | VP35 50 VP35
## VP40 KM034562v1 4390-5894 + | VP40 50 VP40
## GP KM034562v1 5900-8305 + | GP 50 GP
## sGP KM034562v1 5900-8305 + | sGP 50 sGP
## ssGP KM034562v1 5900-8305 + | ssGP 50 ssGP
## VP30 KM034562v1 8288-9740 + | VP30 50 VP30
## VP24 KM034562v1 9885-11518 + | VP24 50 VP24
## L KM034562v1 11501-18282 + | L 50 L
## biotype
## <character>
## NP gene
## VP35 gene
## VP40 gene
## GP gene
## sGP gene
## ssGP gene
## VP30 gene
## VP24 gene
## L gene
## -------
## seqinfo: 1 sequence from gadmor1_he567025 genome# Delete the temporary database
unlink(tmpdb)Please note that complete annotations from UCSC require the genePredToGtf
tool from the UCSC tools base and runs only on Linux. The tool is required
only for building 3’ UTR annotations from UCSC, RefSeq and NCBI, all of which
are being retrieved from the UCSC databases. The next example (full EBOLA virus
annotation, not evaluated) demonstrates how this is done in a Unix based
machine:
# Setup a temporary directory to download files etc.
customDir <- file.path(tempdir(),"test_custom")
dir.create(customDir)
# Convert from GenePred to GTF - Unix/Linux only!
if (.Platform$OS.type == "unix" && !grepl("^darwin",R.version$os)) {
# Download data from UCSC
goldenPath="http://hgdownload.cse.ucsc.edu/goldenPath/"
# Gene annotation dump
download.file(paste0(goldenPath,"eboVir3/database/ncbiGene.txt.gz"),
file.path(customDir,"eboVir3_ncbiGene.txt.gz"))
# Chromosome information
download.file(paste0(goldenPath,"eboVir3/database/chromInfo.txt.gz"),
file.path(customDir,"eboVir3_chromInfo.txt.gz"))
# Prepare the build
chromInfo <- read.delim(file.path(customDir,"eboVir3_chromInfo.txt.gz"),
header=FALSE)
chromInfo <- chromInfo[,1:2]
rownames(chromInfo) <- as.character(chromInfo[,1])
chromInfo <- chromInfo[,2,drop=FALSE]
# Coversion from genePred to GTF
genePredToGtf <- file.path(customDir,"genePredToGtf")
if (!file.exists(genePredToGtf)) {
download.file(
"http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/genePredToGtf",
genePredToGtf
)
system(paste("chmod 775",genePredToGtf))
}
gtfFile <- file.path(customDir,"eboVir3.gtf")
tmpName <- file.path(customDir,paste(format(Sys.time(),"%Y%m%d%H%M%S"),
"tgtf",sep="."))
command <- paste0(
"zcat ",file.path(customDir,"eboVir3_ncbiGene.txt.gz"),
" | ","cut -f2- | ",genePredToGtf," file stdin ",tmpName,
" -source=eboVir3"," -utr && grep -vP '\t\\.\t\\.\t' ",tmpName," > ",
gtfFile
)
system(command)
# Build with the metadata list filled (you can also provide a version)
addCustomAnnotation(
gtfFile=gtfFile,
metadata=list(
organism="eboVir3_test",
source="ucsc_test",
chromInfo=chromInfo
),
db=myDb
)
# Try to retrieve some data
eboGenes <- loadAnnotation(genome="eboVir3_test",refdb="ucsc_test",
type="gene",db=myDb)
eboGenes
}Another example, a sample of the Atlantic cod genome annotation from UCSC.
gtfFile <- system.file(package="sitadela","extdata",
"gadMor1_HE567025.gtf.gz")
chromInfo <- read.delim(system.file(package="sitadela","extdata",
"gadMor1_HE567025.txt.gz"),header=FALSE)
# Build with the metadata list filled (you can also provide a version)
addCustomAnnotation(
gtfFile=gtfFile,
metadata=list(
organism="gadMor1_test",
source="ucsc_test",
chromInfo=chromInfo
),
db=myDb
)## Opening sitadela SQLite database /tmp/RtmppfVC9v/test_anndb/testann.sqlite## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/gadMor1_HE567025.gtf.gz as GTF to make id map## Making id map## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/gadMor1_HE567025.gtf.gz as TxDb## Import genomic features from the file as a GRanges object ... OK
## Prepare the 'metadata' data frame ... OK
## Make the TxDb object ... OK
## Retrieving gene annotation for gadmor1_test from ucsc_test version 20210519 from /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/gadMor1_HE567025.gtf.gz
## Retrieving transcript annotation for gadmor1_test from ucsc_test version 20210519
## Retrieving summarized transcript annotation for gadmor1_test from ucsc_test version 20210519
## Retrieving 3' UTR annotation for gadmor1_test from ucsc_test version 20210519
## 3' UTR annotation for gadmor1_test from ucsc_test version 20210519 is not available in the provided GTF file.
## Retrieving summarized 3' UTR annotation per gene for gadmor1_test from ucsc_test version 20210519
## 3' UTR annotation for gadmor1_test from ucsc_test version 20210519 is not available in the provided GTF file.
## Retrieving summarized 3' UTR annotation per transcript for gadmor1_test from ucsc_test version 20210519
## 3' UTR annotation for gadmor1_test from ucsc_test version 20210519 is not available in the provided GTF file.
## Retrieving exon annotation for gadmor1_test from ucsc_test version 20210519
## Retrieving summarized exon annotation for gadmor1_test from ucsc_test version 20210519
## Retrieving extended exon annotation for gadmor1_test from ucsc_test version 20210519
## Retrieving summarized transcript exon annotation for gadmor1_test from ucsc_test version 20210519# Try to retrieve some data
gadGenes <- loadAnnotation(genome="gadMor1_test",refdb="ucsc_test",
type="gene",db=myDb)
gadGenes## GRanges object with 48 ranges and 4 metadata columns:
## seqnames ranges strand | gene_id gc_content gene_name
## <Rle> <IRanges> <Rle> | <character> <numeric> <character>
## g8912 1 66-6023 + | g8912 50 g8912
## g8913 1 17576-54518 - | g8913 50 g8913
## g8914 1 74456-75028 - | g8914 50 g8914
## g8915 1 98451-108568 - | g8915 50 g8915
## g8916 1 129805-168324 + | g8916 50 g8916
## ... ... ... ... . ... ... ...
## g8955 1 960225-962523 + | g8955 50 g8955
## g8956 1 969370-988129 - | g8956 50 g8956
## g8957 1 989587-1008879 - | g8957 50 g8957
## g8958 1 1018881-1041482 - | g8958 50 g8958
## g8959 1 1044660-1068026 + | g8959 50 g8959
## biotype
## <character>
## g8912 gene
## g8913 gene
## g8914 gene
## g8915 gene
## g8916 gene
## ... ...
## g8955 gene
## g8956 gene
## g8957 gene
## g8958 gene
## g8959 gene
## -------
## seqinfo: 1 sequence from gadmor1_test genome; no seqlengthsAnother example, Armadillo from Ensembl. This should work irrespectively of operating system. We are downloading chromosomal information from UCSC. Again, a small dataset included in the package is included in this vignette. See the commented code below for the full annotation case.
gtfFile <- system.file(package="sitadela","extdata",
"dasNov3_JH569334.gtf.gz")
chromInfo <- read.delim(system.file(package="sitadela",
"extdata","dasNov3_JH569334.txt.gz"),header=FALSE)
# Build with the metadata list filled (you can also provide a version)
addCustomAnnotation(
gtfFile=gtfFile,
metadata=list(
organism="dasNov3_test",
source="ensembl_test",
chromInfo=chromInfo
),
db=myDb
)## Opening sitadela SQLite database /tmp/RtmppfVC9v/test_anndb/testann.sqlite## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/dasNov3_JH569334.gtf.gz as GTF to make id map## Making id map## Importing GTF /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/dasNov3_JH569334.gtf.gz as TxDb## Import genomic features from the file as a GRanges object ... OK
## Prepare the 'metadata' data frame ... OK
## Make the TxDb object ... OK
## Retrieving gene annotation for dasnov3_test from ensembl_test version 20210519 from /tmp/RtmpAwLtxW/Rinst10c7af74fb5b01/sitadela/extdata/dasNov3_JH569334.gtf.gz
## Retrieving transcript annotation for dasnov3_test from ensembl_test version 20210519
## Retrieving summarized transcript annotation for dasnov3_test from ensembl_test version 20210519
## Retrieving 3' UTR annotation for dasnov3_test from ensembl_test version 20210519
## Retrieving summarized 3' UTR annotation per gene for dasnov3_test from ensembl_test version 20210519
## Retrieving summarized 3' UTR annotation per transcript for dasnov3_test from ensembl_test version 20210519
## Retrieving exon annotation for dasnov3_test from ensembl_test version 20210519
## Retrieving summarized exon annotation for dasnov3_test from ensembl_test version 20210519
## Retrieving extended exon annotation for dasnov3_test from ensembl_test version 20210519
## Retrieving summarized transcript exon annotation for dasnov3_test from ensembl_test version 20210519# Try to retrieve some data
dasGenes <- loadAnnotation(genome="dasNov3_test",refdb="ensembl_test",
type="gene",db=myDb)
dasGenes## GRanges object with 49 ranges and 4 metadata columns:
## seqnames ranges strand | gene_id
## <Rle> <IRanges> <Rle> | <character>
## ENSDNOG00000040310 1 39921-57597 + | ENSDNOG00000040310
## ENSDNOG00000026053 1 75778-75866 - | ENSDNOG00000026053
## ENSDNOG00000047749 1 107506-107609 - | ENSDNOG00000047749
## ENSDNOG00000049646 1 118767-167082 - | ENSDNOG00000049646
## ENSDNOG00000019696 1 234318-380483 + | ENSDNOG00000019696
## ... ... ... ... . ...
## ENSDNOG00000031698 1 4891267-5067477 + | ENSDNOG00000031698
## ENSDNOG00000040409 1 4967800-4968430 + | ENSDNOG00000040409
## ENSDNOG00000036092 1 5130036-5232074 - | ENSDNOG00000036092
## ENSDNOG00000050381 1 5345174-5346286 - | ENSDNOG00000050381
## ENSDNOG00000050589 1 5370552-5414125 + | ENSDNOG00000050589
## gc_content gene_name biotype
## <numeric> <character> <character>
## ENSDNOG00000040310 50 SNRPD1 protein_coding
## ENSDNOG00000026053 50 SNORA63 snoRNA
## ENSDNOG00000047749 50 ENSDNOG00000047749 snoRNA
## ENSDNOG00000049646 50 ABHD3 protein_coding
## ENSDNOG00000019696 50 MIB1 protein_coding
## ... ... ... ...
## ENSDNOG00000031698 50 TAF4B protein_coding
## ENSDNOG00000040409 50 ENSDNOG00000040409 protein_coding
## ENSDNOG00000036092 50 ENSDNOG00000036092 protein_coding
## ENSDNOG00000050381 50 ENSDNOG00000050381 lincRNA
## ENSDNOG00000050589 50 ENSDNOG00000050589 lincRNA
## -------
## seqinfo: 1 sequence from dasnov3_test genome; no seqlengthsA quite complete build (with latest versions of Ensembl annotations) would look like (supposing the default annotation database location):
organisms <- list(
hg18=67,
hg19=75,
hg38=101:102,
mm9=67,
mm10=101:102,
rn5=77,
rn6=101:102,
dm3=77,
dm6=101:102,
danrer7=77,
danrer10=91,
danrer11=101:102,
pantro4=90,
pantro5=101:102,
susscr3=89,
susscr11=101:102,
equcab2=94,
equcab3=101:102
)
sources <- c("ensembl","ucsc","refseq","ncbi")
addAnnotation(organisms,sources,forceDownload=FALSE,rc=0.5)The aforementioned complete built can be found here Complete builts will become available from time to time (e.g. with every new Ensembl relrase) for users who do not wish to create annotation databases on their own. Root access may be required (depending on the sitadela library location) to place it in the default location where it can be found automatically.
If for some reason you do not want to build and use an annotation database but
you wish to benefit from the sitadela simple formats nonetheless, or even to
work with an organism that does not yet exist in the database, the
loadAnnotation function will perform all required actions (download and create
a GRanges object) on-the-fly as long as there is an internet connection.
However, the aforementioned function does not handle custom annotations in GTF
files. In that case, you should use the importCustomAnnotation function with
a list describing the GTF file, that is:
metadata <- list(
organism="ORGANISM_NAME",
source="SOURCE_NAME",
chromInfo="CHROM_INFO"
)The above argument can be passed to the importCustomAnnotation call in the
respective position.
For further details about custom annotations on the fly, please check
addCustomAnnotation and importCustomAnnotation functions.
sessionInfo()## R version 4.1.0 (2021-05-18)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.2 LTS
##
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.13-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.13-bioc/R/lib/libRlapack.so
##
## locale:
## [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
## [3] LC_TIME=en_GB LC_COLLATE=C
## [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
## [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
## [9] LC_ADDRESS=C LC_TELEPHONE=C
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
##
## attached base packages:
## [1] stats4 parallel stats graphics grDevices utils datasets
## [8] methods base
##
## other attached packages:
## [1] BSgenome.Mmusculus.UCSC.mm9_1.4.0 BSgenome_1.60.0
## [3] rtracklayer_1.52.0 Biostrings_2.60.0
## [5] XVector_0.32.0 GenomicRanges_1.44.0
## [7] GenomeInfoDb_1.28.0 IRanges_2.26.0
## [9] S4Vectors_0.30.0 BiocGenerics_0.38.0
## [11] sitadela_1.0.0 BiocStyle_2.20.0
##
## loaded via a namespace (and not attached):
## [1] MatrixGenerics_1.4.0 Biobase_2.52.0
## [3] httr_1.4.2 RMySQL_0.10.21
## [5] sass_0.4.0 bit64_4.0.5
## [7] jsonlite_1.7.2 bslib_0.2.5.1
## [9] assertthat_0.2.1 BiocManager_1.30.15
## [11] BiocFileCache_2.0.0 blob_1.2.1
## [13] GenomeInfoDbData_1.2.6 Rsamtools_2.8.0
## [15] yaml_2.2.1 progress_1.2.2
## [17] lattice_0.20-44 pillar_1.6.1
## [19] RSQLite_2.2.7 glue_1.4.2
## [21] digest_0.6.27 Matrix_1.3-3
## [23] htmltools_0.5.1.1 XML_3.99-0.6
## [25] pkgconfig_2.0.3 biomaRt_2.48.0
## [27] bookdown_0.22 zlibbioc_1.38.0
## [29] purrr_0.3.4 BiocParallel_1.26.0
## [31] tibble_3.1.2 KEGGREST_1.32.0
## [33] generics_0.1.0 ellipsis_0.3.2
## [35] withr_2.4.2 cachem_1.0.5
## [37] SummarizedExperiment_1.22.0 GenomicFeatures_1.44.0
## [39] magrittr_2.0.1 crayon_1.4.1
## [41] memoise_2.0.0 evaluate_0.14
## [43] fansi_0.4.2 xml2_1.3.2
## [45] tools_4.1.0 prettyunits_1.1.1
## [47] hms_1.1.0 matrixStats_0.58.0
## [49] BiocIO_1.2.0 lifecycle_1.0.0
## [51] stringr_1.4.0 DelayedArray_0.18.0
## [53] AnnotationDbi_1.54.0 compiler_4.1.0
## [55] jquerylib_0.1.4 rlang_0.4.11
## [57] grid_4.1.0 RCurl_1.98-1.3
## [59] rjson_0.2.20 rappdirs_0.3.3
## [61] bitops_1.0-7 rmarkdown_2.8
## [63] restfulr_0.0.13 DBI_1.1.1
## [65] curl_4.3.1 R6_2.5.0
## [67] GenomicAlignments_1.28.0 knitr_1.33
## [69] dplyr_1.0.6 fastmap_1.1.0
## [71] bit_4.0.4 utf8_1.2.1
## [73] filelock_1.0.2 stringi_1.6.2
## [75] Rcpp_1.0.6 vctrs_0.3.8
## [77] png_0.1-7 dbplyr_2.1.1
## [79] tidyselect_1.1.1 xfun_0.23