Greetings GenBank Users,
GenBank Release 186.0 is now available via FTP from the
National Center for Biotechnology Information (NCBI):
Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 186.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 186.0
Close-of-data for GenBank 186.0 occurred on 10/13/2011. Uncompressed,
the Release 186.0 flatfiles require roughly 518 GB (sequence files only)
or 557 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 426 GB.
Recent statistics for non-WGS, non-CON sequences:
Release Date Base Pairs Entries
185 Aug 2011 130671233801 142284608
186 Oct 2011 132067413372 144458648
Recent statistics for WGS sequences:
Release Date Base Pairs Entries
185 Aug 2011 208315831132 64997137
186 Oct 2011 218666368056 68330215
During the 60 days between the close dates for GenBank Releases 185.0
and 186.0, the non-WGS/non-CON portion of GenBank grew by 1,396,179,571
basepairs and by 2,174,040 sequence records. During that same period,
789,185 records were updated. An average of 49,387 non-WGS/non-CON
records were added and/or updated per day.
Between releases 185.0 and 186.0, the WGS component of GenBank grew by
10,350,536,924 basepairs and by 3,333,078 sequence records.
For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 186.0 and Upcoming Changes) have been appended
below for your convenience.
** Important Notes **
* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.3 of the release notes for
further details.
NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.
Release 186.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.
As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: October 15 2011, 186.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.
A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a Unix or Linux platform, using csh/tcsh :
set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end
Or, if the files are compressed, perhaps:
gzcat $i | head -10 | grep Release
If you encounter problems while ftp'ing or uncompressing Release
186.0, please send email outlining your difficulties to:
info from ncbi.nlm.nih.gov
Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky, Sergey Zhdanov
GenBank
NCBI/NLM/NIH/HHS
1.3 Important Changes in Release 186.0
1.3.1 Organizational changes
The total number of sequence data files increased by 23 with this release:
- the BCT division is now composed of 77 files (+2)
- the CON division is now composed of 152 files (+3)
- the ENV division is now composed of 43 files (+1)
- the EST division is now composed of 450 files (+3)
- the HTC division is now composed of 14 files (-1)
- the HTG division is now composed of 136 files (+1)
- the INV division is now composed of 30 files (-1)
- the PAT division is now composed of 171 files (+3)
- the PLN division is now composed of 51 files (+1)
- the PRI division is now composed of 44 files (+2)
- the TSA division is now composed of 43 files (+8)
- the VRL division is now composed of 19 files (+1)
The number of HTC division files decreased by one because nearly 37,000
high-throughput sequences for human cDNA clones were removed at the
submitter's request.
The number of INV division files decreased by one because WGS project CABG01
has replaced approximately 19,000 (non-WGS) records for the Schistosoma mansoni
genome.
The total number of 'index' files increased by 2 with this release:
- the AUT (author name) index is now composed of 91 files (+2)
1.3.2 New centromere and telomere features
Telomeres and centromeres are essential features of chromosomes and
disrupting their structure affects the viability and life span of an
organism. Centromeric sequence varies from a compact, non-repetitive,
less than 150 base pair region in S. cerevisiae to a highly repetitive
and complex region of several hundred thousands of base pairs in
eukaryote genomes. The sequence at the telomeric ends is unique compared
to the rest of the chromosome and protects the chromosome ends from
recombination, fusion to other chromosomes or degradation by nucleases.
Currently telomere and centromere features may be under-annotated since
there are no specific feature keys for them, hence the INSDC approved
the creation of two new features, which are now allowed as of the
October 15 2011 GenBank Release.
These two features are intended for use when the centromere or telomere
have been actually been sequenced. These two new features are now legal as
of this GenBank Release 186.0 (October 15 2011).
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feature Key centromere
Definition region of biological interest identified as a centromere
and which have been experimentally characterized
Optional qualifiers /centromere_type=<centromere_type>
/citation=[number]
/db_xref="<database>:<identifier>"
/experiment="[CATEGORY:]text"
/inference="[CATEGORY:]TYPE[ (same species)][:EVIDENCE_BASIS]"
/note="text"
/standard_name="text"
Comment the centromere feature describes the interval of DNA
that corresponds to a region where chromatids are held
and a kinetochore is formed
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Qualifier /centromere_type=
Definition type of centromere
Value format point, regional
Example /centromere_type=point
Comment the values are case-insensitive, i.e. both "POINT" and
"point" are valid;
Definitions of the values:
regional : the DNA sequence consists of large arrays of
repetitive DNA, where the sequence of individual repeat
elements is similar but not identical
point : consist of well defined and conserved DNA
sequences that are sufficient to confirm centromere
identity and function
Important Note: The necessity of a /centromere_type qualifier is still under
discussion. If it is determined that the type can *always* be inferred based
on the organism alone (eg, 'point' for S. cerevisiae and 5 other budding yeasts,
'regional' for all other eukaryotes) then /centromere_type will not be
implemented.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feature Key telomere
Definition region of biological interest identified as a telomere
and which have been experimentally characterized
Optional qualifiers /citation=[number]
/db_xref="<database>:<identifier>"
/experiment="[CATEGORY:]text"/note="text"
/inference="[CATEGORY:]TYPE[ (same species)][:EVIDENCE_BASIS]"
/note="text"
/rpt_unit_seq
/rpt_unit_range
/rpt_type
/standard_name="text"
Comment the telomere feature describes the interval of DNA
that corresponds to a specific structure at the end of
the linear eukaryotic chromosome which is required for
the integrity and maintenance of the end; this region
is unique compared to the rest of the chromosome and
represents the physical end of the chromosome;
1.3.3 New assembly_gap feature, and /gap_type and /linkage_evidence qualifiers
Complete genomes are often submitted to the INSDC via a small (or large)
set of independent sequence records, which can be assembled into chromosomes
and/or scaffolds. The CON-division records representing these scaffolds
and chromosomes are usually built using information provided in "AGP files"
provided by the submitter. See:
http://www.ncbi.nlm.nih.gov/genome/assembly/agp/AGP_Update.shtml **
The AGP 2.0 specification includes provisions for a variety of different
gap types, as well as information about whether a gap between two
scaffold or chromosome components is an unspanned gap or a spanned gap.
There is also biological gap-types: telomere, centromere and repeat.
AGP 2.0 also supports terminology to describe the type of evidence used
to establish the linkage connecting the components on either side of a
spanned gap within a scaffold or chromosome. Unfortunately, there is no
mechanism to represent any of this information in the Feature Table.
To address this, the INSDC has decided to implement an assembly_gap
feature, and /gap_type and /linkage_evidence qualifiers, all of which
are now legal as of this October 15 2011 GenBank Release.
The new centromere and telomere features (see Section 1.3.2) should
only be used when the actual sequence of a centromere/telomere has been
determined. If this is not the case, then an assembly_gap feature with
a /gap_type of "centromere" or "telomere" should be used instead to
indicate the locations of centromeres and telomeres within a genome
assembly.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feature Key assembly_gap
Definition gap between two components of a CON-division record that
is part of a genome assembly
Mandatory qualifiers /estimated_length=unknown or <integer>
/gap_type="TYPE"
/linkage_evidence="TYPE" (Note: Mandatory only if the
/gap_type is "within scaffold" or "repeat within scaffold"
For all other types of assembly_gap features, use of the
/linkage_evidence qualifier is invalid.)
Comment the location span of the assembly_gap feature for an unknown
gap is 100 bp, with the 100 bp indicated as 100 "n"'s in the
sequence. Where estimated length is indicated by an integer,
this is indicated by the same number of "n"'s in the sequence.
No upper or lower limit is set on the size of the gap.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Qualifier /gap_type=TYPE
Definition kind of gap connecting components in records of a genome
assembly, or the kind of biological gap in a record that
is part of a genome assembly
Value format "between scaffolds", "within scaffold", "telomere",
"centromere", "short arm", "heterochromatin",
"repeat within scaffold", "repeat between scaffolds"
Example /gap_type="between scaffolds"
/gap_type="within scaffold"
Comment This qualifier is used only for assembly_gap features and
its values are controlled by the AGP Specification version 2.0:
http://www.ncbi.nlm.nih.gov/genome/assembly/agp/AGP_Update.shtml **
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Qualifier /linkage_evidence=TYPE
Definition type of evidence establishing linkage across an assembly_gap.
Only allowed to be used with assembly_gap features that have
a /gap_type value of "within scaffold" or "repeat within scaffold"
Value format "paired-ends", "align genus", "align xgenus", "align trnscpt",
"within clone", "clone contig", "map", "strobe", "unspecified"
Example /linkage_evidence="paired-ends"
/linkage_evidence="within clone"
Comment This qualifier is used only for assembly_gap features and its
values are controlled by the AGP Specification version 2.0
http://www.ncbi.nlm.nih.gov/genome/assembly/agp/AGP_Update.shtml **
** The current URL for the AGP 2.0 specification is temporary. When a permanent link
becomes available, the Feature Table will be updated accordingly.
Although these new features and qualifiers are allowed as of Oct 15 2011,
it is unlikely that they will begin to appear on sequence records before
December 1st. However, users should begin preparing for them now, to prevent
processing problems later.
1.3.4 Changes in the content of index files
As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics from January 2005
seemed to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.
The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.
Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.
The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.
These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:
a) Cease support of the 'index' file products altogether.
b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.
If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:
info from ncbi.nlm.nih.gov
Our apologies for any inconvenience that these changes may cause.
1.3.5 GSS File Header Problem
GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.
There is thus a discrepancy between the filenames and file headers for
103 of the GSS flatfiles in Release 186.0. Consider gbgss146.seq :
GBGSS1.SEQ Genetic Sequence Data Bank
October 15 2011
NCBI-GenBank Flat File Release 186.0
GSS Sequences (Part 1)
87119 loci, 64001369 bases, from 87119 reported sequences
Here, the filename and part number in the header is "1", though the file
has been renamed as "146" based on the number of files dumped from the other
system. We hope to resolve this discrepancy at some point, but the priority
is certainly much lower than many other tasks.
1.4 Upcoming Changes
There are no changes scheduled for implementation in the February 2012
GenBank Release.
