JBrowse 1.11.1 has been released, with a number of bug fixes and some nice improvements to Alignments2 tracks. Most of the Alignments2 improvements were thought up by Keiran Raine at Sanger, thanks Keiran! Thanks also to Cris Lawrence and Richard Hayes, who were helpful in finding some other bugs and opportunities for improvements.
Alignments2 tracks now include right-click menu items to view the
location of an alignment's mate pair or next segment in a popup or
a new tab. Thanks to Keiran Raine for suggesting this (issue #406).
Alignments2 tracks now draw gaps and deletions in reads regardless
of zoom level, as long as the alignment is at least 3 pixels wide
in the display. Thanks to Keiran Raine for pointing out the need
for this (issue #403).
Added support for a histograms.max variable for
CanvasFeatures-based tracks that can be used to manually set the
max value of a histogram display. Thanks to Keiran Raine for
pointing out the need for this.
Added support for drawing clip markers (with their color set by
histograms.clip_marker_color) in CanvasFeatures-based
tracks. Thanks to Keiran Raine for pointing out the need for this
(issue #402).
Canvas-based feature tracks now try to draw histograms, if
available, when the data backend throws a data-overflow error (like
when the BAM backend exceeds the chunkSizeLimit). Thanks to Keiran
Raine for motivating this (issue #405).
Make it easier to set JS loading baseUrl by moving it into the
initial dojo configuration. Thanks to Jillian Rowe for pointing
out the need for this.
Enhanced new text-based config syntax to support arrays of values
in a list like:
[trackMetadata]
sources =
data/mymeta.csv
data/more_meta.csv
Bug fixes
When a number is typed into the location box, JBrowse first checks
if it is the name of a feature in the names index, and only
interprets it as a coordinate if it is not found in the names
index. Thanks to Richard Hayes for pointing this out (issue #407).
Fixed bug that caused client-side GFF3 tracks to appear as
"Loading" forever if the GFF3 is malformed (like malformed GFF3
files that are opened with the File->Open tool).
Fixed bug in which no default value for
maxFeatureSizeForUnderlyingRefSeq was being set, which made
default feature detail popups try to fetch and display a feature's
underlying reference sequence even if it is way too large, unless
the variable was set explicitly in the configuration.
JBrowse now shows a more understandable error message when trying
to open an uncompressed BAM file. Thanks to Keiran Raine for
pointing this out (issue #404).
Fixed jbrowse.conf faceted track selector configuration examples
not working as written. Thanks to Cris Lawrence for pointing this
out.
Fixed a bug in which right-clicking on feature labels in an
HTMLFeatures-based track did not bring up the right-click menu for
a feature. Thanks to Cris Lawrence for pointing this out (issue #408).
Firstly, JBrowse has added** a third track selector type** alongside the existing Simple (small) and Faceted (large) track selectors **called Hierarchical**. It looks like the old Simple track selector (which was the default), except it pays attention to the "category" key in a track's metadata, and if it is present, it organizes the tracks into nested, collapsible panes based on that. Also, instead of the drag-and-drop paradigm used by the Simple track selector, the Hierarchical track selector turns tracks on and off by just checking and unchecking the box next to each track. I think the checkboxes are probably easier for new users to understand, as well, the Hierarchical track selector is now enabled by default. Those that prefer the old Simple selector can always turn it on by setting trackSelector.type to "Simple" in the jbrowse_conf.json file or the new jbrowse.conf file, which brings me to my next item.
Secondly, JBrowse now has a additional text configuration format that is designed to be easier to hand edit, and can coexist and interoperate with existing JSON configuration files. Users of GBrowse will find the syntax very familiar, since its design borrows heavily from GBrowse's configuration syntax. Here's a side-by-side comparison of what you would write in the old JSON and new text configuration formats to make JBrowse use the old Simple track selector.
jbrowse_conf.json
jbrowse.conf
"trackSelector": {
"type": "Simple"
}
[trackSelector]
type = Simple
It's easier to configure tracks in the new format, as well. Here's a typical BAM Alignments2 track configuration in both formats, for comparison.
[tracks.volvox-sorted-bam]
storeClass = JBrowse/Store/SeqFeature/BAM
urlTemplate = ../../raw/volvox/volvox-sorted.bam
type = JBrowse/View/Track/Alignments2
key = My BAM track
And for the final headline feature of this release, JBrowse CanvasFeatures, CanvasVariants, and Alignments2 tracks can now be configured to show** feature density histograms or coverage plots** when zoomed too far out to display individual features. They can take this quantitative data from** any type of JBrowse data store** (including REST stores), but most users would want to use a BigWig store. One caveat: this isn't a full-featured quantitative display like a Wiggle/XYPlot track, so quantitative values need to all be greater than 0, and all you can really change about how it looks is its color. Here's an example in the new text configuration syntax of an Alignments2 track that will display either alignments from a BAM file or a coverage plot from a BigWig file, depending on zoom level.
Of course, there are many smaller improvements and bug fixes. Read on for the complete release notes.
Release 1.11.0
Major improvements
Introduced density/coverage histogram support for CanvasFeatures,
CanvasVariants, and Alignments2 tracks. These track types now
support an optional histograms configuration subsection that can
contain a definition for a second datastore that holds quantitative
data (usually either coverage depth or feature density) to be
displayed when zoomed further out than featureScale (or if
featureScale is not set, the scale determined by the store's
feature density divided by maxFeatureScreenDensity). Thanks to
Richard Hayes for pushing hard for this feature.
Added a new "Hierarchical" track selector that shows tracks in a
hierarchy of collapsible categories, which is now the default track
selector. To assign categories and subcategories to your tracks,
set category or metadata.category attributes on each configured
tracks in your trackList.json. Thanks to the many users who have
requested this at one time or another.
JBrowse now supports a new plaintext configuration format that
users of GBrowse will find very familiar, since it is designed to
be very similar to it. This syntax is also much easier to
hand-write than JSON. The JSON configuration syntax is not going
away, and will continue to be supported.
Thanks to Erik Derohanian for the original implementation of this
configuration adaptor, and Richard Hayes and Keiran Raine for
motivating the work to polish and more fully integrate it.
Variables in configuration files can now be based on the contents
of other variables. For example, setting
will try to include a configuration file located at
"/some/custom/path/conf.json". Interpolation is done as the final
step in configuration loading, so variables can come from anywhere
in the configuration.
When JBrowse is started, if there are no reference sequences found
in the default dataRoot, but the dataset selector is configured,
JBrowse shows a simple list of links to available datasets instead
of the "Congratulations, JBrowse is on the web" page. Thanks to
Saulo Aflitos for the idea and its initial implementation.
For users wishing to convert existing JSON configuration files to
the new format, there is a new script, bin/json2conf.pl, that
does a fair job. Run bin/json2conf.pl -? for details on how to
use it.
Major performance and scalability improvements for
generate-names.pl. Now uses a different algorithm that is faster
and more scalable than before, and no longer relies on BerkeleyDB
for temporary storage. This should also alleviate the need to run
generate-names.pl with --safeMode in Perl 5.10 and earlier. In
fact, the --safeMode argument to generate-names.pl no longer has
any effect. Thanks to Cris Lawrence for pointing out the
continuing need for more scalability.
Minor improvements
Detail popups for CanvasVariants and HTMLVariants tracks now
display the reference sequence itself instead of just "ref" in
genotype displays. Thanks to Cris Lawrence for requesting this.
Added a "save as FASTA" button to default feature detail popups
that downloads a FASTA file with the displayed piece of reference
sequence (issue #299).
chunkSizeLimit for VCF files now defaults to 1 MiB. It used to
be 15 MiB, which was really far too big for browsers to handle.
Added support for a --nameAttributes argument to
flatfile-to-json.pl that takes a comma-separated list of feature
attributes to index for name searching and completions, or 'none'
to not make names searchable.
Added support for a nameAttributes variable in biodb-to-json.pl
track configurations that can be set to an array of feature
attribute names to to index for name searching and completions, or
'none' to not make names searchable.
Add a --category argument to bin/wig-to-json.pl that can be used
to set the metadata.category of a track.
Bug fixes
Fixed a bug in NCList data backed in which feature histograms were
often calculated very incorrectly.
Fixed a bug in the VCF data backend that caused not all VCF
features to be shown in some files at some zoom levels.
JBrowse 1.10.12 has been released, and we are asking all users of JBrowse 1.10.7 1.10.10 and higher to please upgrade to this release as soon as possible.
The story behind this is a bit embarassing: I discovered today that JBrowse releases 1.10.7 through 1.10.10 and 1.10.11 shipped with usage analytics reporting disabled. This is a little piece of code that tracks how many installations and users of JBrowse there are, and the data it provides is absolutely crucial for the JBrowse project, because we use it to try to convince funding agencies that JBrowse is a project worth funding.
So, yes ... this release has analytics reporting re-enabled. There are also a few other minor improvements and bug fixes. Please upgrade to this ASAP if you are using JBrowse 1.10.7, 1.10.8, 1.10.9, 1.10.10 or 1.10.11, because we really need that analytics data.
**Edit (Dec 11): **realized just now that only 1.10.10 and 1.10.11 were affected. Forgot to account for branching and merging last night.
bam-to-json.pl and flatfile-to-json.pl now support a
--metadata argument that can add a metadata stanza to track
configurations they generate.
Multi-valued attributes in default feature detail popups are
displayed as a string of boxes, each containing a value, to avoid
misinterpretation. Thanks to Cris Lawrence for pointing out the
need for this.
Bug fixes
Re-enabled usage analytics reporting, which had been disabled by a
stray piece of debugging code since the 1.10.7 release.
Fixed a bug in which the tooltip in canvas-based feature tracks
would sometimes display incorrect label or description text.
JBrowse 1.10.11 has been released, with many bug fixes and better support for spliced alignments in SNPCoverage tracks.
(Download disabled. Please download the[ 1.10.12 release](http://jbrowse.org/jbrowse-1-10-12/ "JBrowse 1.10.12 maintenance release") or later instead.)
Minor improvements
Made the sequence track's "zoom in to see sequence" placeholder
take up less vertical space. Thanks to Scott Cain for pointing out
that making it be the same height as it will eventually be when
zoomed in to base level is silly.
By default highlighting features after searching for them by name
is now turned off. Set the highlightSearchedRegions top-level
conf variable to true to turn this back on. Turns out, most
people seem not to like this behavior. Thanks to Gregg Helt and
Cris Lawrence for pointing this out.
SNPCoverage tracks now correctly display "skipped" regions in
alignments, such as those produced by TopHat. Thanks to Josh
Orvis, Gustavo Cerquiera, and others for reminders that this was
still an issue.
SNPCoverage tracks now provide per-strand counts of "reference"
reads at each position, like they already were providing for
reads with mismatches.
SNPCoverage tracks now accept a mismatchScale configuration
variable that sets the viewing scale (i.e. zoom level, pixels per
bp) above which base-level mismatches will be drawn. Defaults to
1/10. Making this value larger can speed up SNPCoverage tracks for
high-coverage data at the cost of needing to zoom in further to see
mismatches.
setup.sh now uses curl for downloading things instead of wget,
since curl is more widely available. Thanks to Keiran Raine for
suggesting and implementing this (issue #393).
Bug fixes
Fixed a bug in which generate-names.pl would sometimes report the
incorrect number of hashing bits in verbose output, and would
sometimes use the number of hashing bits for an existing index even
if that index was being regenerated. Thanks to Richard Hayes for
pointing out the incorrect log output.
Fix generate-names.pl crashing on some older versions of Perl with
an error like Bareword "POSIX::O_RDONLY" not allowed while "strict subs" in use. Thanks to Chris Childers for pointing this out.
Fix setup.sh failing on some older versions of Perl. It now runs
generate-names.pl with the --safeMode flag.
Fixed a bug where the value display in SNPCoverage tracks would
sometimes report "NaN%" for the reference when no reads cover a
region.
Fixed a bug in which activating rubberband zooming using the SHIFT
key while in highlighting mode would cause all the tracks to be
dragged when attempting to highlight a region afterward. Thanks to
Erik Derohanian for pointing out and fixing this (issue #387).
Fixed the location of the dojo/nls directory in release zipfiles.
Was erroneously in src/nls, supposed to be src/dojo/nls. Thanks to
Matt Bomhoff for pointing this out.
The in-memory GFF3 parser now copes with a missing newline at the
end of a GFF3 file. Previously, the last line was ignored if it
did not end with a newline character. Thanks to Colin Davenport
for pointing this out (issue #394).
JBrowse 1.10.10 has been released, with new compression support in generate-names.pl, configurable inter-track spacing, and quite a few bug fixes.
(Download disabled. Please download the[ 1.10.12 release](http://jbrowse.org/jbrowse-1-10-12/ "JBrowse 1.10.12 maintenance release") or later instead.)
Minor improvements
Spacing between tracks is now configurable by setting
view.trackPadding in the configuration. Thanks to Chenchen Zhu
for suggesting this (issue #377).
If reference sequences are defined, but no tracks are yet added,
JBrowse will start normally instead of going to the
"Congratulations, JBrowse is on the web" page.
generate-names.pl now supports a --compress option that
compresses the name index files to save server disk space. Thanks
to Richard Hayes for pointing out the need for this (issue #378).
Bug fixes
generate-names.pl now uses IO::Uncompress::Gunzip instead of
PerlIO::gzip to read compressed VCF files. This fixes a bug in
which only the first few hundred names in a VCF were indexed.
Thanks to Cris Lawrence for pointing this out (issue #380).
Fixed a bug in which generate-names.pl would crash if run with
--incremental and no existing names index. Thanks to Richard
Hayes for pointing this out (issue #379).
Fixed a bug in which generate-names.pl would sometimes choose the
wrong number of hash bits when performing incremental updates,
leading to the old data being lost. Thanks to Richard Hayes for
lots of help troubleshooting this.
Fixed bug where other tracks are visible underneath pinned tracks
when the display is scrolled down. Thanks to Ed Lee for pointing
this out.
Fixed a bug in which tooltip does not hide after the mouse leaves a
Wiggle track in Safari 5 and 6. Thanks to Charles Girardot for
pointing this out.
Fixed a bug in which the main.css file for plugins was not being
correctly loaded in some installations. Thanks to Matt Bomhoff for
pointing this out.
generate-names.pl now supports a --incremental or -i option
that adds names to an existing index. Thanks to Richard Hayes for
reminding me that this wasn't done yet (issue #373).
Added a lower-performance, but more backward-compatible indexing
backend to generate-names.pl that can be activated by passing the
--safeMode command-line argument. The recent performance
improvements to generate-names.pl have apparently tickled some bugs
that are present in some installations. Thanks to Josie Reinhardt
and GitHub user raj76 for their continued help troubleshooting
this. If you find that name indexing is not working correctly, try
running it again with --safeMode, and report to the mailing list
if it helps.
Added support for using the JBrowse in-memory GFF3 adaptor to
display web-accessible GFF3 files directly. See
docs/tutorial/data_files/volvox.gff3.conf for an example
configuration. Thanks to David Goodstein and Richard Hayes for
motivating this.
Bug fixes
flatfile-to-json.pl now depends on the latest
Bio::GFF3::LowLevel::Parser 1.8, which fixes a bug in which
features with no ID, Parent, or Derives_from attributes were not
being included in parsed data. Thanks to Gwendoline Andres for
pointing this out.
Tweak BioPerl-handling code in biodb-to-json.pl to hopefully work
better with BioSQL backends. Thanks to Brian Osborne for pointing
this out.
Added a new navigateTo action usable for customizing feature
left-clicks and right-click menus. Thanks to Scott Cain for
requesting this.
Added a new feature_range_cache option for the REST data backend.
If set to true, the REST backend will more aggressively cache
ranges of feature data. Thanks to Daniel Troesser for pointing out
the need for this (issue #369).
maker2jbrowse now, by default, runs generate-names.pl to
generate names indexes. Also added a --no_names_index option to
turn this off. Thanks to Josie Reinhardt for making me notice this
was missing.
Tweaked generate-names.pl default indexing parameters to
emphasize indexing speed more. Now defaults to a smaller average
file size for the on-disk JSON files it produces, which is much
faster to generate and write (up to 8 or 10x faster). However, the
on-disk index is about 2-2.5x larger overall.
Removed support for the --refids command-line argument to
prepare-refseqs.pl, which has probably never really worked.
Improved prepare-refseqs.pl support for Bio::DB::Das::Chado
database backends. Thanks to Gwendoline Andres for helping
troubleshoot this.
Bug fixes
Fixed generate-names.pl making incorrect name indexes when using 16
or more bits of hashing (when the number of index entries exceeds
about 4 million). Thanks to Josie Reinhardt for helping
troubleshoot this (issue #370).
The faceted track selector no longer refuses to display track
unique labels even if they are explicitly included in the
displayColumns setting.
Added an optional stats/regionFeatureDensities endpoint to the
REST API that makes it possible to provide binned feature counts
that HTMLFeatures tracks can use to display feature histograms.
Thanks to Stuart Watt and Daniel Troesser for pointing out the need
for this (issue #365).
Bug fixes
Fixed a bad bug introduced in 1.10.6 in which FASTA files with line
lengths longer than the configured chunk size were not correctly
formatted. Thanks to Jean-Jack Riethoven for pointing this out
(issue #363).
Fixed a bug introduced in 1.10.6 in which prepare-refseqs.pl would
not respect the --noseq command-line option when using --fasta.
Fixed bug in which name or seq_id attributes are required for
reference sequence features. Thanks to Daniel Troesser for
pointing this out.
Fixed a bug that may have prevented some types of VCF files from
being displayed (error message referring to inheritedFilters).
Fixed a bug in which "Zoom in to see feature" in Sequence tracks
would wrap downwards below track at some zoom levels and window
widths.
Fixed an off-by-one error in UTR attributes manufactured by the
impliedUTRs mechanism of the ProcessedTranscript glyph. Thanks
to Ben Booth for pointing this out (issue #362).
Fixed a bug in which score attributes of features were not being
recorded by biodb-to-json.pl. Thanks to HongKee Moon for
pointing this out (issue #364).
Removed a stray use of Carp::Always in GFF3-processing Perl code
that may have caused problems in some installations.
Made the impliedUTRs option for ProcessedTranscript and Gene
glyph still attempt to create UTRs if only '''one''' of the UTRs is
missing from a transcript. Thanks to Ben Booth for pointing out
the need for this.
Bug fixes
Fixed bug in which the mouseover value displays for Wiggle and
SNPCoverage tracks would not always be hidden when the mouse leaves
the track.
Fixed a bad bug that prevented fixed-scale Wiggle and SNPCoverage
tracks from displaying. Thanks to Jean-Jack Riethoven and Michael
Axtell for pointing this out.
Greatly improved the speed and reduced the memory footprint when
running prepare-refseqs.pl with the --fasta and --gff options.
Added an impliedUTRs option to the ProcessedTranscript and Gene
glyphs for CanvasFeatures tracks. Thanks to Ben Booth for pointing
out the need for this (issue #348).
Upgraded flatfile-to-json.pl to use a new version of
Bio::GFF3::LowLevel::Parser for GFF3 parser, which has a lookback
buffer limit that makes it easier to parse large GFF3 files that do
not contain enough '###' directives.
Further improved the memory footprint and speed of
generate-names.pl. Thanks to Richard Hayes for his continued help
with testing improvements and reporting problems.
Removed explicit dependency on GD::Image, which is only used by the
old tiled-image generation demo code. This will make dependency
installation easier for many people, at a (very) small cost to
backward compatibility.
Bug fixes
Fixed a bad bug that prevented combination tracks from working.
Thanks to Harry Yoo for pointing this out (issue #351).
Fixed a bug in which one of the temporary files used by
generate-names.pl was not being created in the correct location.
Thanks to Richard Hayes for testing this.
Fixed a bug in which generate-names.pl could sometimes crash when
run with the -v (verbose) switch, or in setup.sh.
Fixed odd behavior when entering coordinate ranges in the location
box under Safari. Thanks to Keiran Raine for pointing this out
(issue #341).