Upgrading
dep
support, drop Go 1.9
supportFeatures:
nsqadmin
: switch to go:embed for static assetsnsqd
: ability to skip ephemeral topics/channels in statsd outputnsqd
: ability to configure auth endpoint path (thanks @tufanbarisyildirim)Context
to use stdlib context
nsqd
: minimize allocations on message send (thanks @imxyb)uniq
(thanks @donutloop)nsqd
: ability to omit memory stats from /stats
(thanks @creker)nsqd
: only update internal RDY
count for client when it changes (thanks @andyxning)nsqd
: bump go-diskqueue
(interface change) (thanks @bitpeng)sort.Ints
over sort.Sort
(thanks @lelenanam)nsqd
: configurable queue scan worker pool (thanks @andyxning)nsqd
: don't buffer messages when --mem-queue-size=0
(thanks @bitpeng)nsqd
: support separate broadcast ports for TCP and HTTP (thanks @shyam-king)Bugs:
nsqadmin
: fix graphite key for ephemeral topics/channelslog_level
support in configuration file (thanks @edoger)nsqd
: handle SIGTERMnsqadmin
: fix --proxy-graphite
support (thanks @fanlix)nsqlookupd
: fix incorrect error message for HTTP listener (thanks @TangDH03)nsqd
: fix live lock for high churn ephemeral topic/channel reconnections (thanks @slayercat)nsswitch.conf
to ensure go resolver uses /etc/hosts
firstnsqd
: improve error message when --data-path
does not exist (thanks @mdh67899)nsqadmin
: fix channel delete, fix source-maps in Firefox (thanks @avtrifanov)nsqadmin
: fix nodes list with ipv6 addresses (thanks @andyxning)Upgrading
nsqd
: removed support for old metadata scheme used in v0.3.8 and earlier
dep
support still present for now, but deprecatedFeatures:
nsqd
: add --max-channel-consumers
(default unlimited) (thanks @andyxning)nsqd
: add --min-output-buffer-timeout
(default 25ms) to limit how low a timeout a consumer can request
--max-output-buffer-timeout
to 30 seconds (lower timeout, more cpu usage)nsqd
: add topic total message bytes to stats (thanks @andyxning)nsqd
: add flag to adjust default --output-buffer-timeout
(thanks @andyxning)nsqd
: add random load balancing for authd requests (thanks @shenhui0509)nsqd
: include client TLS cert CommonName in authd requestsnsq_to_file
: include topic/channel in most log messagesnsq_to_file
: add --log-level
and --log-prefix
flagsnsq_to_file
: big refactor, more robust file switching and syncing and error handlingnsqd
: add param to /stats
endpoint to allow skipping per-client stats (much faster if many clients)nsqadmin
, nsq_stat
: use include_clients
param for /stats
for a big speedup for big clustersnsq_to_file
: support for separate working directory with --work-dir
(thanks @mccutchen)nsqadmin
: add --base-path
flag (thanks @blinklv)nsq_to_http
: add --header
flag (thanks @alwindoss)nsqd
: add producer client tcp connections to stats (thanks @sparklxb)nsq_to_file
: new flag --sync-interval
(default same as previous behavior, 30 seconds) (thanks @alpaker)Bugs:
nsqd
: close connections that don't send "magic" header (thanks @JoseFeng)nsqd
: exit on all fatal Accept() errors - restart enables better recovery for some conditions (thanks @mdh67899)nsqd
, nsqlookupd
, nsqadmin
: refactor LogLevel, general refactor to better exit on all fatal errorsnsqadmin
: switch to using judwhite/go-svc
like nsqd
and nsqadmin
donsqadmin
: fix clients count and channel total message rate (new bugs introduced in this cycle)nsqd
, nsqlookupd
, nsqadmin
: fix http error response unreliable json serializationnsqlookupd
: fix orphaned ephemeral topics in registration DBnsqd
: fix topic message mis-counting if there are backend write errors (thanks @SwanSpouse)nsqlookupd
: optimize /nodes
endpoint, much better for hundreds of nsqd (thanks @andyxning)GOFLAGS
to BLDFLAGS
in Makefile
now that GOFLAGS
is automatically used by gonsqadmin
: eslint reported fixes/cleanupsUpgrading from 1.0.0-compat: Just a few backwards incompatible changes:
nsq_pubsub
utilitynsqd
flags --msg-timeout
and --statsd-interval
only take duration strings
nsqd
: http /mpub
endpoint binary
param interprets "0" or "false" to mean text mode
The previous release, version "1.0.0-compat", was curiously-named to indicate an almost (but not quite) complete transition to a 1.0 api-stable release line. Confusingly, this follow-up release which completes the transition comes more than a year later. Because there have been a fair number of changes and improvements in the past year, an additional minor version bump seems appropriate.
Features:
nsqd
: add memory stats to http /stats
response (thanks @sparklxb)nsqd
, nsqlookupd
, nsqadmin
: add --log-level
option (deprecating --verbose
) (thanks @antihax)nsqd
, nsqlookupd
, nsqadmin
: logging refactor to use log levels everywherensqadmin
: X-Forwarded-User
based "admin" permission (thanks @chen-anders)nsqd
: add topic/channel filter to /stats
, use in nsqadmin
and nsq_stat
for efficiency (thanks @andyxning)nsq_to_file
: refactor/cleanupnsq_to_nsq
: support multiple --topic
flags (thanks @jlr52)nsq_tail
: support multiple --topic
flags (thanks @soar)nsqd
, nsqadmin
: update internal http client with new go http.Transport
features (keepalives, timeouts, dualstack)
nsqadmin
, nsqd
, nsqlookupd
dep
(replacing gpm
) (thanks @judwhite)nsqd
: better memory usage when messages have different sizes (thanks @andyxning)nsqd
: optimize random channel selection in queueScanLoop (thanks @vearne)nsqd
: buffer and spread statsd udp sends (avoid big burst of udp, less chance of loss)nsqlookupd
: optimize for many producers (thousands) (thanks @andyxning)nsqd
: new topic can be unlocked faster after creationnsqadmin
: update JS depsBugs:
nsqadmin
: fix missing channels in topic listto_nsq
: fix divide-by-zero issue when --rate
not specified (thanks @adamweiner)nsqd
: clamp requeue timeout to range instead of dropping connection (thanks @tsholmes)nsqd
: fix race when client subscribes to ephemeral topic or channel while it is being cleaned up (reported by @slayercat)nsqd
: fix deflate level handlingnsqd
: fix channel shutdown flush racensq_to_file
: fix connection leaks when using --topic-pattern
(thanks @jxskiss)nsqlookupd
: close connection when magic read fails (thanks @yc90s)nsqd
: fix some races getting ChannelStats (thanks @daroot)nsqd
: fix e2e timings config example, add range validation (thanks @protoss-player)nsq_tail
: logging to stderr (only nsq messages to stdout)nsq_to_http
: fix http connect/request timeout flags (thanks @kamyanskiy)nsqd
: fix possible lookupd-identify-error busy-loop (reported by @andyxning)nsqadmin
: fix typo "Delfate" in connection attrs list (thanks @arussellsaw)nsqd
: fix loading metadata with messages queued on un-paused topic with multiple channels (thanks @michaelyou)nsqlookupd
: exit with error when failed to listen on ports (thanks @stephens2424)nsqadmin
: fix html escaping for large_graph url inside javascriptUpgrading from 0.3.8: Numerous backwards incompatible changes:
nsqd
features removed:
/{m,}put
(use /{m,}pub
)/{create,delete,empty,pause,unpause}_{topic,channel}
(use /{topic,channel}/<operation>
)--max-message-size
flag (use --max-msg-size
)V2
protocol IDENTIFY
command short_id
and long_id
properties (use client_id
, hostname
, and user_agent
)/stats
HTTP response name
property (use client_id
)nsqlookupd
features removed:
/{create,delete}_{topic,channel}
(use /{topic,channel}/<operation>
)/tombstone_topic_producer
(use /topic/tombstone
)nsqadmin
features removed:
--template-dir
flag (not required, templates are compiled into binary)--use-statsd-prefixes
flag (use --statsd-counter-format
and --statsd-gauge-format
)nsq_stat
--status-every
flag (use --interval
)--reader-opt
on all binaries that had this flag (use --consumer-opt
)nsq_to_file
--gzip-compression
flag (use --gzip-level
)nsq_to_http
--http-timeout
and --http-timeout-ms
flags (use --http-connect-timeout
and --http-request-timeout
)nsq_to_http
--round-robin
flag (use --mode=round-robin
)nsq_to_http
--max-backoff-duration
flag (use --consumer-opt=max_backoff_duration,X
)nsq_to_http
--throttle-fraction
flag (use --sample=X
)nsq_to_nsq
--max-backoff-duration
flag (use --consumer-opt=max_backoff_duration,X
)nsqd
--worker-id
deprecated in favor of --node-id
(to be fully removed in subsequent release)This is a compatibility release that drops a wide range of previously deprecated features (#367) while introducing some new deprecations (#844) that we intend to fully remove in a subsequent 1.0 release.
Of note, all of the pre-1.0 HTTP endpoints (and response formats) are gone. Any clients or tools that use these endpoints/response formats won't work with this release. These changes have been available since 0.2.29 (released in July of 2014). Clients wishing to forwards-compatibly upgrade can either use the new endpoints or send the following header:
Accept: application/vnd.nsq version=1.0
Also, many command line flags have been removed — in almost all cases an alternative is available with a (hopefully) more obvious name. These changes have the same affect on config file option names.
On Linux, this release will automatically migrate nsq.<worker-id>.dat
named metadata files to
nsq.dat
in a way that allows users to seamlessly downgrade from this release back to 0.3.8, if
necessary. A subsequent release will clean up these convenience symlinks and observe only
nsq.dat
. See the discussion in #741 and the changes #844 for more details.
Performance wise, #741 landed which significantly reduces global contention on internal message ID generation, providing a ~1.75x speed improvement on multi-topic benchmarks.
Finally, a number of minor issues were resolved spanning contributions from 9 community members! Thanks!
Features:
alpine
base image for official Docker container (thanks @kenjones-cisco)nsqadmin:
/config
API (thanks @kenjones-cisco)nsqadmin
, nsq_stat
, nsq_to_file
, nsq_to_http
: HTTP client connect/request timeouts (thanks @kenjones-cisco)to_nsq
: add --rate
message throttling optionnsqd
: per-topic message IDs (multi-topic pub benchmarks up to ~1.75x faster)nsqd
, nsqlookupd
, nsqadmin
: add --log-prefix
option (thanks @ploxiln)nsqd
: deprecate --worker-id
for --node-id
and drop ID from nsqd.dat
file (thanks @ploxiln)Bugs:
nsqlookupd
: properly close TCP connection in IOLoop
(thanks @JackDrogon)nsqdmin
: fix root CA verification (thanks @joshuarubin)nsq_to_file
: require --topic
or --topic-pattern
(thanks @judwhite)nsqadmin
: fix handling of IPv6 broadcast addresses (thanks @magnetised)nsqd
: fix requeue and deferred message accounting (thanks @sdbaiguanghe)nsqd
: switch to golang/snappy to fix snappy deadlocknsqd
: fix default --broadcast-address
and error when nsqlookupd
reqs fail (thanks @ploxiln @stephensearles)nsqd
: prevent panic in binary /mpub
(thanks @yangyifeng01)nsqadmin
: allow ctrl/meta+click to open a new tabnsqd
: check for exit before requeingUpgrading from 0.3.7: Binaries contain no backwards incompatible changes.
This release fixes a critical regression in 0.3.7
that could result in message loss when
attempting to cleanly shutdown nsqd
by sending it a SIGTERM
. The expected behavior was for it
to flush messages in internal buffers to disk before exiting. See #757 and #759 for more details.
A few performance improvements landed including #743, which improves channel throughput by ~17%, and #740, which reduces garbage when reading messages from disk.
We're now stripping debug info, reducing binary size, in the official binary downloads and Windows
binaries are now bundled with the appropriate .exe
extension (#726 and #751).
Features:
nsqd
: remove channel messagePump
nsqd
: reduce garbage when reading from diskqueue (thanks @dieterbe)Bugs:
nsqd
: properly handle SIGTERM
(thanks @judwhite)go-options
nsqd
: diskqueue sync count on both read/writensqadmin
: make rate
column work without --proxy-graphite
(thanks @ploxiln).exe
extension to Windows binaries (thanks @ploxiln)nsqadmin
: fix connected duration > 1hr
Upgrading from 0.3.6: Binaries contain no backwards incompatible changes.
This release has been built with Go 1.6.
Highlights include the various work done to reduce nsqd
lock contention, significantly improving
the impact of high load on the /stats
endpoint, addressing issues with timeouts and failures
in nsqadmin
(#700, #701, #703, #709).
Thanks to @judwhite, nsqd
and nsqlookupd
now natively support being run as a Windows service
(#718). We're also now publishing official Windows releases.
nsqd
will now flock
its data directory on linux, preventing two nsqd
from running
simultaneously pointed at the same path (#583).
On the bugfix side, the most noteworthy change is that nsqd
will now correctly reset health state
on a successful backend write (#671).
Features:
nsqd
: reduce lock contention (thanks @zachbadgett @absolute8511)nsqd
/nsqlookupd
: support running as a windows service (thanks @judwhite)nsqd
: support enabling/disabling block profile via HTTP (thanks @absolute8511)nsqd
: support POST
/debug/pprof/symbol
(thanks @absolute8511)nsqadmin
: add flags for formatting statsd keys (thanks @kesutton)nsqd
: flock
--data-path
on linuxnsqd
: optimize GUID generation (thanks @ploxiln)Bugs:
nsqd
: fix max size accounting in diskqueue
(thanks @judwhite)nsqd
: reset health on successful backend write (thanks @judwhite)nsqd
: prevent OOM when reading from nsqlookupd
peerUpgrading from 0.3.5: Binaries contain no backwards incompatible changes.
We've adopted the Contributor Covenant 1.2 Code of Conduct (#593). Help us keep NSQ open and inclusive by reading and following this document.
We closed a few longstanding issues related to nsqadmin
, namely (#323, et al.) converting it to
an API and single-page app (so that it is much easier to develop), displaying fine-grained errors
(#421, #657), and enabling support for --tls-required
configurations (#396).
For nsqd
, we added support for deferred publishing aka DPUB
(#293), which allows a producer to
specify a duration of time to delay initial delivery of the message. We also addressed performance
issues relating to large numbers of topics/channels (#577) by removing some per-channel goroutines
in favor of a centralized, periodic, garbage collection approach.
In order to provide more flexibility when deploying NSQ in dynamically orchestrated topologies,
nsqd
now supports the ability to configure nsqlookupd
peers at runtime via HTTP (#601),
eliminating the need to restart the daemon.
As part of the large nsqadmin
refactoring, we took the opportunity to cleanup the internals for
all of the daemon's HTTP code paths (#601, #610, #612, #641) as well as improving the test suite
so that it doesn't leave around temporary files (#553).
Features:
nsqadmin
: convert to API / single-page appnsqadmin
: expand notification contextnsqd
: add deferred pub (DPUB
)nsqd
: drop per-channel queue workers in favor of centralized queue GCnsqlookupd
: improve registration DB performance (thanks @xiaost)nsqd
: HTTP endpoints to dynamically configure nsqlookupd
peersnsqd
: support for filtering /stats
to topic/channel (thanks @chrusty)nsqd
: clarify help text for --e2e-processing-latency-percentile
--{consumer,producer}-opt
to nsq.ConfigFlag
Bugs:
nsqadmin
: update statsd
prefix to stats.counters
nsqadmin
: display upstream/partial errorsnsqdamin
/nsqd
: support for --tls-required
nsqd
: ignore benign EOF errorsnsqd
: GUID error handling / catch errors if GUID goes backwards (thanks @mpe)nsqd
: fix valid range for --worker-id
nsqd
: fix benchmarks (thanks @Dieterbe)nsqd
: enforce diskqueue min/max message size (thanks @twmb)Upgrading from 0.3.3: Binaries contain no backwards incompatible changes.
This is another quick bug fix release to address the broken nsqadmin
binary in the distribution
(see #578).
WARNING: please upgrade to v0.3.5
to address the broken nsqadmin
binary.
Upgrading from 0.3.3: Binaries contain no backwards incompatible changes.
This is a quick bug fix release to fix the outdated go-nsq
dependency in v0.3.3
for the bundled utilities (see 6e8504e
).
WARNING: please upgrade to v0.3.5
to address the outdated go-nsq
dependency for the
bundled utilities and the broken nsqadmin
binary.
Upgrading from 0.3.2: Binaries contain no backwards incompatible changes.
This release is primarily a bug fix release after cleaning up and reorganizing the codebase.
nsqadmin
is now importable, which paves the way for completing #323. The bundled utilities
received a few feature additions and bug fixes (mostly from bug fixes on the go-nsq
side).
Features:
nsqadmin
: re-org into importable packagensq_to_{nsq,http}
: add epsilon-greedy
mode (thanks @twmb)nsqd
: adds start_time
to /stats
(thanks @ShawnSpooner)nsq_to_http
: accept any 200
response as success (thanks @mikedewar)nsq_to_http
: read entire request body (thanks @imgix)/internal
package re-org (thanks @cespare)Bugs:
nsqd
: don't persist metadata upon startup (thanks @xiaost)nsqd
: do not print EOF
error when client closes cleanly (thanks @twmb)nsqd
: fix --tls-required=tcp-https
with --tls-client-auth-policy
(thanks @twmb)NOTE: the bundled utilities are built against go-nsq
v1.0.4
and include all of
those features/fixes.
Upgrading from 0.3.1: Binaries contain no backwards incompatible changes however as of this release we've updated our official Docker images.
We now provide a single Docker image nsqio/nsq
that includes all of the NSQ binaries. We did this for several reasons, primarily because the
tagged versions in the previous incarnation were broken (and did not actually pin to a version!).
The new image is an order of magnitude smaller, weighing in around 70mb.
In addition, the impetus for this quick release is to address a slew of reconnect related bug fixes
in the utility apps (nsq_to_nsq
, nsq_to_file
, etc.), for details see the go-nsq
v1.0.3
release notes.
Features:
Bugs:
#ephemeral
topic deletion issuesUpgrading from 0.3.0: No backwards incompatible changes.
This release contains minor bug fixes and feature additions.
There are a number of functionality improvements to the nsq_stat
and nsq_to_file
helper
applications (and general support for #ephemeral
topics, broken in 0.2.30
).
Additionally, the TLS options continue to improve with support for setting --tls-min-version
and
a work-around for a bug relating to TLS_FALLBACK_SCSV
(to be fixed in Go
1.5).
Features:
--status-every
in favor of --interval
--count
option (thanks @nordicdyno)--tls-min-version
and set TLS max version to 1.2--tls-required
can be disabled for HTTP / add --tls-min-version
(thanks @twmb)<PID>
to filename and rotation by size/interval (thanks @xiaost)BytesToBase10
(thanks @iand)Bugs:
#ephemeral
topic deletion issuesdiskqueue
atomic rename on Windows (thanks @allgeek)output_buffer_*
resolved settings in IDENTIFY
response (thanks @tj)Upgrading from 0.2.31: No backwards incompatible changes.
This release includes a slew of bug fixes and few key feature additions.
The biggest functional change is that nsqd
no longer decrements its RDY
count for clients. This
means that client libraries no longer have to periodically re-send RDY
. For some context, nsqd
already provided back-pressure due to the fact that a client must respond to messages before
receiving new ones. The decremented RDY
count only made the implementation of the server and
client more complex without additional benefit. Now the RDY
command can be treated as an "on/off"
switch. For details see #404 and the associated changes in nsqio/go-nsq#83 and nsqio/pynsq#98.
The second biggest change (and oft-requested feature!) is #ephemeral
topics. Their behavior
mirrors that of channels. This feature is incredibly useful for situations where you're using
topics to "route" messages to consumers (like RPC) or when a backlog of messages is undesirable.
There are now scripts in the bench
directory that automate the process of running a distributed
benchmark. This is a work-in-progress, but it already provides a closer-to-production setup and
therefore more accurate results. There's much work to do here!
A whole bunch of bugs were fixed - notably all were 3rd-party contributions! Thanks!
#ephemeral
topicsRDY
/ send RDY
before FIN
/REQ
nsqd
diskqueue
sync strategiesnsq_to_file
(thanks @lxfontes)nsqd
IOLoop
logging (thanks @rexposadas)Bugs:
nsqd
Notify()
goroutines have exited prior to shutdown (thanks @allgeek)diskqueue
syncs at end of benchmarks (thanks @Dieterbe)TestPauseMetadata
(thanks @allgeek)dist.sh
(thanks @losinggeneration)nsqadmin
checks stats for missing topic (thanks @jnewmano)nsqd
decodes corrupt message data (thanks @twmb)nsq_to_nsq
/nsq_to_http
(thanks @leshik)go get
compatibility (thanks @adams-sarah)Upgrading from 0.2.30: No backwards incompatible changes.
This release includes a few key changes. First, we improved feedback and back-pressure when nsqd
writes to disk. Previously this was asynchronous and would result in clients not knowing that their
PUB
had failed. Interestingly, this refactoring improved performance of PUB
by 41%, by removing
the topic's goroutine responsible for message routing in favor of N:N
Go channel communication.
For details see #437.
@paddyforan contributed official Dockerfiles that are now built automatically via Docker Hub. Please begin to use (and improve these) as the various older images we had been maintaining will be deprecated.
The utility apps deprecated the --reader-opt
flag in favor of --consumer-opt
and nsq_to_nsq
and to_nsq
received a --producer-opt
flag, for configuring details of the connection publishing
to nsqd
. Additionally, it is now possible to configure client side TLS certificates via
tls_cert
and tls_key
opts.
As usual, we fixed a few minor bugs, see below for details.
New Features / Enhancements:
nsqd
: diskqueue error feedback/backpressure (thanks @boyand)nsqd
, nsqlookupd
, nsqadmin
(thanks @paddyforan)--consumer-opt
alias for --reader-opt
and
add --producer-opt
to nsq_to_nsq
(also support configuration
of tls_cert
and tls_key
)nsqd
: improve IOLoop error messages (thanks @rexposadas)Bugs:
nsqd
: fixed statsd GC stats reporting (thanks @jphines)nsqd
: improve handling/documentation of --worker-id
(thanks @bschwartz)nsqd
: IDENTIFY
should respond with materialized msg_timeout
(thanks @visionmedia)Upgrading from 0.2.29: No backwards incompatible changes.
IMPORTANT: this is a quick bug-fix release to address a panic in nsq_to_nsq
and
nsq_to_http
, see #425.
New Features / Enhancements:
nsqadmin
/nsqd
: expose TLS connection statensq_to_nsq
/nsq_to_file
: display per-destination-address timingsBugs:
nsq_to_nsq
/nsq_to_file
: fix shared mutable state panicUpgrading from 0.2.28: No backwards incompatible changes.
This release includes a slew of new features and bug fixes, with contributions from 8 members of the community, thanks!
The most important new feature is authentication (the AUTH
command for nsqd
), added in #356.
When nsqd
is configured with an --auth-http-address
it will require clients to send the AUTH
command. The AUTH
command body is opaque to nsqd
, it simply passes it along to the configured
auth daemon which responds with well formed JSON, indicating which topics/channels and properties
on those entities are accessible to that client (rejecting the client if it accesses anything
prohibited). For more details, see the spec or the
nsqd
guide.
Additionally, we've improved performance in a few areas. First, we refactored in-flight handling in
nsqd
to reduce garbage creation and improve baseline performance 6%. End-to-end processing
latency calculations are also significantly faster, thanks to improvements in the
perks
package.
HTTP response formats have been improved (removing the redundant response wrapper) and cleaning up some of the endpoint namespaces. This change is backwards compatible. Clients wishing to move towards the new response format can either use the new endpoint names or send the following header:
Accept: application/vnd.nsq version=1.0
Other changes including officially bumping the character limit for topic and channel names to 64
(thanks @svmehta), making the REQ
timeout limit configurable in nsqd
(thanks @AlphaB), and
compiling static asset dependencies into nsqadmin
to simplify deployment (thanks @crossjam).
Finally, to_nsq
was added to the suite of bundled apps. It takes a stdin stream and publishes to
nsqd
, an extremely flexible solution (thanks @matryer)!
As for bugs, they're mostly minor, see the pull requests referenced in the section below for details.
New Features / Enhancements:
to_nsq
for piping stdin to NSQ (thanks @matryer)nsqadmin
: embed external static asset dependencies (thanks @crossjam)user_agent
nsqd
: improve in-flight message handling (6% faster, GC reduction)nsqd
: introduce AUTH
go-nsq
dependencies (GC reduction)nsqd
: make REQ
timeout limit configurable (thanks @AlphaB)nsqd
: bump perks dependency (E2E performance improvement, see 25086e4
)Bugs:
nsqd
: fix statsd GC time reportingnsqd
: fix double TOUCH
and use of client's configured msg timeoutnsqadmin
: fix HTTPS warning (thanks @juliangruber)nsqlookupd
: fix race on last update timestampnsqd
: properly handle empty FIN
nsqd
: fix IDENTIFY
msg_timeout
response (thanks @visionmedia)nsq_to_file
: set proper permissions on new directories (thanks @bschwartz)nsqd
: fix windows diskqueue filenames (thanks @politician)Upgrading from 0.2.27: No backwards incompatible changes. We've deprecated the short_id
and long_id
options in the IDENTIFY
command in favor of client_id
and hostname
, which
more accurately reflect the data typically used.
This release includes a few important new features, in particular enhanced nsqd
TLS support thanks to a big contribution by @chrisroberts.
You can now require that clients negotiate TLS with --tls-required
and you can configure a
client certificate policy via --tls-client-auth-policy
(require
or require-verify
):
require
- the client must offer a certificate, otherwise rejectedrequire-verify
- the client must offer a valid certificate according to the default CA or
the chain specified by --tls-root-ca-file
, otherwise rejectedThis can be used as a form of client authentication.
Additionally, nsqd
is now structured such that it is importable in other Go applications
via github.com/nsqio/nsq/nsqd
, thanks to @kzvezdarov.
Finally, thanks to @paddyforan, nsq_to_file
can now archive multiple topics or
optionally archive all discovered topics (by specifying no --topic
params
and using --lookupd-http-address
).
New Features / Enhancements:
nsq_to_file
can archive many topics (thanks @paddyforan)nsqd
TLS client certificate verification policy, ability
to require TLS, and HTTPS support (thanks @chrisroberts)nsqd
importable (github.com/nsqio/nsq/nsqd
) (thanks @kzvezdarov)IDENTIFY
options (replace short_id
and long_id
with
client_id
and hostname
)nsq_to_file
filenames (thanks @jsocol)nsq_stat
Bug Fixes:
nsq.Reader
connection close/exit, this
impacts the utilities packaged with the NSQ binary
distribution such as nsq_to_file
, nsq_to_http
,
nsq_to_nsq
and nsq_tail
.Upgrading from 0.2.26: No backwards incompatible changes. We deprecated --max-message-size
in favor of --max-msg-size
for consistency with the rest of the flag names.
IMPORTANT: this is another quick bug-fix release to address an issue in nsqadmin
where templates
were incompatible with older versions of Go (pre-1.2).
nsqadmin
template compatibility (and formatting)nsqadmin
behavior when E2E stats are disablednsqadmin
INVALID_ERROR
on node page tombstone linknsqd
client metadata race condition and test flakinessnsqd
test races (run w/ -race
and GOMAXPROCS=4
) deprecate --max-message-size
Upgrading from 0.2.25: No backwards incompatible changes.
IMPORTANT: this is a quick bug-fix release to address a regression identified in 0.2.25
where
statsd
prefixes were broken when using the default (or any) prefix that contained a %s
for
automatic host replacement.
nsqd
--statsd-prefix
when using %s
host replacementUpgrading from 0.2.24: No backwards incompatible changes.
This release adds several commonly requested features.
First, thanks to @elubow you can now configure your clients to sample
the stream they're subscribed to. To read more about the details of the implementation see #286 and
the original discussion in #223. Eric also contributed an improvement to nsq_tail
to add
the ability to tail the last N
messages and exit.
We added config file support (TOML) for
nsqd
, nsqlookupd
, and nsqadmin
- providing even more deployment flexibility. Example configs
are in the contrib
directory. Command line arguments override the equivalent option in the config
file.
We added the ability to pause a topic (it is already possible to pause individual channels). This functionality stops all message flow from topic to channel for all channels of a topic, queueing at the topic level. This enables all kinds of interesting possibilities like atomic channel renames and trivial infrastructure wide operations.
Finally, we now compile the static assets used by nsqadmin
into the binary, simplifying
deployment. This means that --template-dir
is now deprecated and will be removed in a future
release and you can remove the templates you previously deployed and maintained.
New Features / Enhancements:
IDENTIFY
option to sample a % of messagesnsqd
, nsqlookupd
, and nsqadmin
nsqadmin
binarynsq_tail
support for -n #
to get recent # messagesIDENTIFY
attributes in nsqadmin
(sample rate, TLS, compression)nsqadmin
`nsq_to_nsq
JSON message filtering optionsUpgrading from 0.2.23: No backwards incompatible changes. However, as you'll see below, quite a
few command line flags to the utility apps (nsq_to_http
, nsq_to_file
, nsq_to_http
) were
deprecated and will be removed in the next release. Please use this release to transition over to
the new ones.
NOTE: we are now publishing additional binaries built against go1.2
The most prominent addition is the tracking of end-to-end message processing percentiles. This
measures the amount of time it's taking from PUB
to FIN
per topic/channel. The percentiles are
configurable and, because there is some overhead in collecting this data, it can be turned off
entirely. Please see the section in the docs for
implementation details.
Additionally, the utility apps received comprehensive support for all configurable reader options (including compression, which was previously missing). This necessitated a bit of command line flag cleanup, as follows:
--gzip-compression
in favor of --gzip-level
--verbose
in favor of --reader-opt=verbose
--throttle-fraction
in favor of --sample
--http-timeout-ms
in favor of --http-timeout
(which is a
duration flag)--verbose
in favor of --reader-opt=verbose
--max-backoff-duration
in favor of
--reader-opt=max_backoff_duration=X
--verbose
in favor of --reader-opt=verbose
--max-backoff-duration
in favor of
--reader-opt=max_backoff_duration=X
New Features / Enhancements:
Upgrading from 0.2.22: No backwards incompatible changes.
We now use godep in order to achieve reproducible builds with pinned
dependencies. If you're on go1.1+ you can now just use godep get github.com/nsqio/nsq/...
.
This release includes nsqd
protocol compression feature negotiation.
Snappy and Deflate are
supported, clients can choose their preferred format.
--statsd-prefix
can now be used to modify the prefix for the statsd
keys generated by nsqd
.
This is useful if you want to add datacenter prefixes or remove the default host prefix.
Finally, this release includes a "bug" fix that reduces CPU usage for nsqd
with many clients by
choosing a more reasonable default for a timer used in client output buffering. For more details
see #236.
New Features / Enhancements:
Bug Fixes:
Upgrading from 0.2.21: message timestamps are now officially nanoseconds. The protocol docs
always stated this however nsqd
was actually sending seconds. This may cause some compatibility
issues for client libraries/clients that were taking advantage of this field.
This release also introduces support for TLS feature negotiation in nsqd
. Clients can optionally
enable TLS by using the appropriate handshake via the IDENTIFY
command. See #227.
Significant improvements were made to the HTTP publish endpoints and in flight message handling to reduce GC pressure and eliminate memory abuse vectors. See #242, #239, and #245.
This release also includes a new utility nsq_to_nsq
for performant, low-latency, copying of an NSQ
topic over the TCP protocol.
Finally, a whole suite of debug HTTP endpoints were added (and consolidated) under the
/debug/pprof
namespace. See #238, #248, and #252. As a result nsqd
now supports direct
profiling via Go's pprof
tool, ie:
$ go tool pprof --web http://ip.address:4151/debug/pprof/heap
New Features / Enhancements:
nsqadmin
single node view (with GC/mem graphs)nsq_to_nsq
utility for copying a topic over TCPnsq_to_http
takes --content-type
flag (thanks @michaelhood)nsqadmin
displays tombstoned topics in the /nodes
list/mput
Bug Fixes:
tombstone_topic_producer
action in nsqadmin
missing node infonsqd
metadata on topic/channel changesnsqlookupd
/nsqadmin
would display inactive nodes in /nodes
listnsq_to_file
that caused empty filesUpgrading from 0.2.20: there are no backward incompatible changes in this release.
This release introduces a significant new client feature as well as a slew of consistency and recovery improvements to diskqueue.
First, we expanded the feature negotiation options for clients. There are many cases where you want
different output buffering semantics from nsqd
to your client. You can now control both
output buffer size and the output buffer timeout via new fields in the IDENTIFY
command. You can
even disable output buffering if low latency is a priority.
You can now specify a duration between fsyncs via --sync-timeout
. This is a far better way to
manage when the process fsyncs messages to disk (vs the existing --sync-every
which is based on #
of messages). --sync-every
is now considered a deprecated option and will be removed in a future
release.
Finally, 0.2.20
introduced a significant regression in #176 where a topic would not write messages
to its channels. It is recommended that all users running 0.2.20
upgrade to this release. For
additional information see #217.
New Features / Enhancements:
Bug Fixes:
Upgrading from 0.2.19: there are no backward incompatible changes in this release.
This release adds a couple of convenient features (such as adding the ability to empty a topic) and continues our work to reduce garbage produced at runtime to relieve GC pressure in the Go runtime.
nsqd
now has two new flags to control the max value clients can use to set their heartbeat
interval as well as adjust a clients maximum RDY count. This is all set/communicated via IDENTIFY
.
nsqadmin
now displays nsqd
-> nsqlookupd
connections in the "nodes" view. This is useful for
visualizing how the topology is connected as well as situations where --broadcast-address
is being
used incorrectly.
nsq_to_http
now has a "host pool" mode where upstream state will be adjusted based on
successful/failed requests and for failures, upstreams will be exponentially backed off. This is an
incredibly useful routing mode.
As for bugs, we fixed an issue where "fatal" client errors were not actually being treated as fatal.
Under certain conditions deleting a topic would not clean up all of its files on disk. There was a
reported issue where the --data-path
was not writable by the process and this was only discovered
after message flow began. We added a writability check at startup to improve feedback. Finally.
deferred_count
was being sent as a counter value to statsd, it should be a gauge.
New Features / Enhancements:
Bug Fixes:
Upgrading from 0.2.18: there are no backward incompatible changes in this release.
This release is a small release that introduces one major client side feature and resolves one critical bug.
nsqd
clients can now configure their own heartbeat interval. This is important because as of
0.2.18
all clients (including producers) received heartbeats by default. In certain cases
receiving a heartbeat complicated "simple" clients that just wanted to produce messages and not
handle asynchronous responses. This gives flexibility for the client to decide how it would like
behave.
A critical bug was discovered where emptying a channel would leave client in-flight state inconsistent (it would not zero) which limited deliverability of messages to those clients.
New Features / Enhancements:
Bug Fixes:
Upgrading from 0.2.17: all V2 clients of nsqd now receive heartbeats (previously only clients that subscribed would receive heartbeats, excluding TCP producers).
Upgrading from 0.2.16: follow the notes in the 0.2.17 changelog for upgrading from 0.2.16.
Beyond the important note above regarding heartbeats this release includes nsq_tail
, an extremely
useful utility application that can be used to introspect a topic on the command line. If statsd is
enabled (and graphite in nsqadmin
) we added the ability to retrieve rates for display in
nsqadmin
.
We resolved a few critical issues with data consistency in nsqlookupd
when channels and topics are
deleted. First, deleting a topic would cause that producer to disappear from nsqlookupd
for all
topics. Second, deleting a channel would cause that producer to disappear from the topic list in
nsqlookupd
.
New Features / Enhancements:
Bug Fixes:
Upgrading from 0.2.16: IDENTIFY and SUB now return success responses (they previously only responded to errors). The official Go and Python libraries are forwards/backwards compatible with this change however 3rd party client libraries may not be.
Upgrading from 0.2.15: in #132 deprecations in SUB were removed as well as support for the old,
line oriented, nsqd
metadata file format. For these reasons you should upgrade to 0.2.16
first.
New Features / Enhancements:
Bug Fixes:
Upgrading from 0.2.15: there are no backward incompatible changes in this release.
However, this release introduces the IDENTIFY
command (which supersedes sending
metadata along with SUB
) for clients of nsqd
. The old functionality will be
removed in a future release.
.