#
32e86a82 |
|
24-Nov-2023 |
Warner Losh <imp@FreeBSD.org> |
sbin: Automated cleanup of cdefs and other formatting Apply the following automated changes to try to eliminate no-longer-needed sys/cdefs.h includes as well as now-empty blank lines in a row. Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/ Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/ Remove /\n+#if.*\n#endif.*\n+/ Remove /^#if.*\n#endif.*\n/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/ Sponsored by: Netflix
|
#
1d386b48 |
|
16-Aug-2023 |
Warner Losh <imp@FreeBSD.org> |
Remove $FreeBSD$: one-line .c pattern Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
|
#
136a9bb4 |
|
01-Jun-2023 |
Elyes Haouas <ehaouas@noos.fr> |
hastd: Fix typos Signed-off-by: Elyes Haouas <ehaouas@noos.fr> Reviewed by: imp Pull Request: https://github.com/freebsd/freebsd-src/pull/653
|
#
4d846d26 |
|
10-May-2023 |
Warner Losh <imp@FreeBSD.org> |
spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch up to that fact and revert to their recommended match of BSD-2-Clause. Discussed with: pfg MFC After: 3 days Sponsored by: Netflix
|
#
1de7b4b8 |
|
27-Nov-2017 |
Pedro F. Giffuni <pfg@FreeBSD.org> |
various: general adoption of SPDX licensing ID tags. Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. No functional change intended.
|
#
974a1085 |
|
01-Sep-2014 |
Ed Schouten <ed@FreeBSD.org> |
Add lock annotations to the threading API used by hastd. Approved by: pjd@
|
#
9c539971 |
|
10-Dec-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
Send wakeup to threads waiting on empty queue before releasing the lock to decrease spurious wakeups. Submitted by: davidxu MFC after: 2 weeks
|
#
d685f88b |
|
10-Dec-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
In remote_send_thread, if sending a request fails don't take the request back from the receive queue -- it might already be processed by remote_recv_thread, which lead to crashes like below: (primary) Unable to receive reply header: Connection reset by peer. (primary) Unable to send request (Connection reset by peer): WRITE(954662912, 131072). (primary) Disconnected from kopusha:7772. (primary) Increasing localcnt to 1. (primary) Assertion failed: (old > 0), function refcnt_release, file refcnt.h, line 62. Taking the request back was not necessary (it would properly be processed by the remote_recv_thread) and only complicated things. MFC after: 2 weeks
|
#
8f04423f |
|
10-Dec-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
Add some macros to make the code more readable (no functional chages). MFC after: 2 weeks
|
#
5d69ed53 |
|
10-Dec-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
For memsync replication, hio_countdown is used not only as an indication when a request can be moved to done queue, but also for detecting the current state of memsync request. This approach has problems, e.g. leaking a request if memsynk ack from the secondary failed, or racy usage of write_complete, which should be called only once per write request, but for memsync can be entered by local_send_thread and ggate_send_thread simultaneously. So the following approach is implemented instead: 1) Use hio_countdown only for counting components we waiting to complete, i.e. initially it is always 2 for any replication mode. 2) To distinguish between "memsync ack" and "memsync fin" responses from the secondary, add and use hio_memsyncacked field. 3) write_complete() in component threads is called only before releasing hio_countdown (i.e. before the hio may be returned to the done queue). 4) Add and use hio_writecount refcounter to detect when write_complete() can be called in memsync case. Reported by: Pete French petefrench ingresso.co.uk Tested by: Pete French petefrench ingresso.co.uk MFC after: 2 weeks
|
#
6b66c350 |
|
26-Oct-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
Make hastctl list command output current queue sizes. Reviewed by: pjd MFC after: 1 month
|
#
d03a08e5 |
|
26-Oct-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
Merging local and remote bitmaps must be protected by hr_amp lock. This is believed to fix hastd crashes, which might occur during synchronization, triggered by the failed assertion: Assertion failed: (amp->am_memtab[ext] > 0), function activemap_write_complete, file activemap.c, line 351. MFC after: 1 week
|
#
a818a4ff |
|
19-Sep-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
When updating the map of dirty extents, most recently used extents are kept dirty to reduce the number of on-disk metadata updates. The sequence of operations is: 1) acquire the activemap lock; 2) update in-memory map; 3) if the list of keepdirty extents is changed, update on-disk metadata; 4) release the lock. On-disk updates are not frequent in comparison with in-memory updates, while require much more time. So situations are possible when one thread is updating on-disk metadata and another one is waiting for the activemap lock just to update the in-memory map. Improve this by introducing additional, on-disk map lock: when in-memory map is updated and it is detected that the on-disk map needs update too, the on-disk map lock is acquired and the on-memory lock is released before flushing the map. Reported by: Yamagi Burmeister yamagi.org Tested by: Yamagi Burmeister yamagi.org Reviewed by: pjd Approved by: re (marius) MFC after: 2 weeks
|
#
1c1310ee |
|
19-Sep-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
Use cv_broadcast() instead of cv_signal() when waking up threads waiting on an empty queue as the queue may have several consumers. Before the fix the following scenario was possible: 2 threads are waiting on empty queue, 2 threads are inserting simultaneously. The first inserting thread detects that the queue is empty and is going to send the signal, but before it sends the second thread inserts too. When the first sends the signal only one of the waiting threads receive it while the other one may wait forever. The scenario above is is believed to be the cause of the observed cases, when ggate_recv_thread() was getting stuck on taking free request, while the free queue was not empty. Reviewed by: pjd Tested by: Yamagi Burmeister yamagi.org Approved by: re (marius) MFC after: 2 weeks
|
#
6e87c151 |
|
26-Apr-2013 |
Ed Schouten <ed@FreeBSD.org> |
Use C11 <stdatomic.h> instead of our non-standard <machine/atomic.h>. Reviewed by: pjd
|
#
2adbba66 |
|
25-Feb-2013 |
Mikolaj Golub <trociny@FreeBSD.org> |
Add i/o error counters to hastd(8) and make hastctl(8) display them. This may be useful for detecting problems with HAST disks. Discussed with and reviewed by: pjd MFC after: 1 week
|
#
d6e636c9 |
|
17-Feb-2013 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
- Add support for 'memsync' mode. This is the fastest replication mode that's why it will now be the default. - Bump protocol version to 2 and add backward compatibility for version 1. - Allow to specify hosts by kern.hostid as well (in addition to hostname and kern.hostuuid) in configuration file. Sponsored by: Panzura Tested by: trociny
|
#
c66ee1b3 |
|
04-Jul-2012 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Make use of GEOM Gate direct reads feature. This allows HAST to serve reads with native speed of the underlying provider. There are three situations when direct reads are not used: 1. Data is being synchronized and synchronization source is the secondary node, which means secondary node has more recent data and we should read from it. 2. Local read failed and we have to try to read from the secondary node. 3. Local component is unavailable and all I/O requests are served from the secondary node. Sponsored by: Panzura, http://www.panzura.com MFC after: 1 month
|
#
4c71d263 |
|
03-Jun-2012 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Simplify the code by using snprlcat(). MFC after: 3 days
|
#
f7371578 |
|
05-Feb-2012 |
Mikolaj Golub <trociny@FreeBSD.org> |
If a local write request is from the synchronization thread, when it is synchronizing data that is out of date on the local component, we should not send G_GATE_CMD_DONE acknowledge to the kernel. This fixes the issue, observed in async mode, when on synchronization from the remote component the worker terminated with "G_GATE_CMD_DONE failed" error. Reported by: Artem Kajalainen <artem kayalaynen ru> Reviewed by: pjd MFC after: 1 week
|
#
2b2cb418 |
|
05-Feb-2012 |
Mikolaj Golub <trociny@FreeBSD.org> |
Fix the regression introduced in r226859: if the local component is out of date BIO_READ requests got lost instead of being sent to the remote component. Reviewed by: pjd MFC after: 1 week
|
#
2b1b224d |
|
10-Jan-2012 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
For functions that return -1 on failure check exactly for -1 and not for any negative number. MFC after: 3 days
|
#
4b85a12f |
|
07-Jan-2012 |
Ulrich Spörlein <uqs@FreeBSD.org> |
Spelling fixes for sbin/
|
#
dfb1aece |
|
06-Jan-2012 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
fork(2) returns -1 on failure, not some random negative number. MFC after: 3 days
|
#
07ebc362 |
|
27-Oct-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Implement 'async' mode for HAST. MFC after: 3 days
|
#
3f5bce18 |
|
27-Oct-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Minor cleanups. MFC after: 3 days
|
#
43b8675b |
|
27-Oct-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Reduce indentation. MFC after: 3 days
|
#
5a58d22a |
|
27-Oct-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Improve comment so it doesn't suggest race is possible, but that we handle the race. MFC after: 3 days
|
#
1212a85c |
|
27-Oct-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Monor cleanups. MFC after: 3 days
|
#
8a34134a |
|
27-Oct-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Delay resuid generation until first connection to secondary, not until first write. This way on first connection we will synchronize only the extents that were modified during the lifetime of primary node, not entire GEOM provider. MFC after: 3 days
|
#
e3feec94 |
|
28-Sep-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Correct typo. MFC after: 3 days
|
#
12daf727 |
|
28-Sep-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
If the underlying provider doesn't support BIO_FLUSH, log it only once and don't bother trying in the future. MFC after: 3 days
|
#
518dd4c0 |
|
28-Sep-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
After every activemap change flush disk's write cache, so that write reordering won't make the actual write to be committed before marking the coresponding extent as dirty. It can be disabled in configuration file. If BIO_FLUSH is not supported by the underlying file system we log a warning and never send BIO_FLUSH again to that GEOM provider. MFC after: 3 days
|
#
be1143ef |
|
27-Sep-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
No need to wrap pjdlog functions around with KEEP_ERRNO() macro. MFC after: 3 days
|
#
09c2e843 |
|
27-Sep-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Correct two mistakes when converting asserts to PJDLOG_ASSERT()/PJDLOG_ABORT(). MFC after: 3 days
|
#
adca96f8 |
|
12-Jul-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
Fix indentation. Approved by: pjd (mentor)
|
#
d9f039e0 |
|
28-Jun-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
Check the returned value of activemap_write_complete() and update matadata on disk if needed. This should fix a potential case when extents are cleared in activemap but metadata is not updated on disk. Suggested by: pjd Approved by: pjd (mentor)
|
#
ba2a8224 |
|
17-Jun-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
In HAST we use two sockets - one for only sending the data and one for only receiving the data. In r220271 the unused directions were disabled using shutdown(2). Unfortunately, this broke automatic receive buffer sizing, which currently works only for connections in ETASBLISHED state. It was a root cause of the issue reported by users, when connection between primary and secondary could get stuck. Disable the code introduced in r220271 until the issue with automatic buffer sizing is not resolved. Reported by: Daniel Kalchev <daniel@digsys.bg>, danger, sobomax Tested by: Daniel Kalchev <daniel@digsys.bg>, danger Approved by: pjd (mentor) MFC after: 1 week
|
#
a01a750f |
|
29-May-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
If READ from the local node failed we send the request to the remote node. There is no use in doing this for synchronization requests. Approved by: pjd (mentor) MFC after: 1 week
|
#
3db86c39 |
|
23-May-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Keep statistics on number of BIO_READ, BIO_WRITE, BIO_DELETE and BIO_FLUSH requests as well as number of activemap updates. Number of BIO_WRITEs and activemap updates are especially interesting, because if those two are too close to each other, it means that your workload needs bigger number of dirty extents. Activemap should be updated as rarely as possible. MFC after: 1 week
|
#
0cddb12f |
|
14-May-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Currently we are unable to use capsicum for the primary worker process, because we need to do ioctl(2)s, which are not permitted in the capability mode. What we do now is to chroot(2) to /var/empty, which restricts access to file system name space and we drop privileges to hast user and hast group. This still allows to access to other name spaces, like list of processes, network and sysvipc. To address that, use jail(2) instead of chroot(2). Using jail(2) will restrict access to process table, network (we use ip-less jails) and sysvipc (if security.jail.sysvipc_allowed is turned off). This provides much better separation. MFC after: 1 week
|
#
ac0401e3 |
|
20-Apr-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
When we become primary, we connect to the remote and expect it to be in secondary role. It is possible that the remote node is primary, but only because there was a role change and it didn't finish cleaning up (unmounting file systems, etc.). If we detect such situation, wait for the remote node to switch the role to secondary before accepting I/Os. If we don't wait for it in that case, we will most likely cause split-brain. MFC after: 1 week
|
#
06cbf549 |
|
19-Apr-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Scenario: - We have two nodes connected and synchronized (local counters on both sides are 0). - We take secondary down and recreate it. - Primary connects to it and starts synchronization (but local counters are still 0). - We switch the roles. - Synchronization restarts but data is synchronized now from new primary (because local counters are 0) that doesn't have new data yet. This fix this issue we bump local counter on primary when we discover that connected secondary was recreated and has no data yet. Reported by: trociny Discussed with: trociny Tested by: trociny MFC after: 1 week
|
#
02dfe972 |
|
02-Apr-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Declare directions for sockets between primary and secondary. In HAST we use two sockets - one for only sending the data and one for only receiving the data. MFC after: 1 month
|
#
2a49afac |
|
02-Apr-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Handle the problem described in r220264 by using GEOM GATE queue of unlimited length. This should fix deadlocks reported by HAST users. MFC after: 1 week
|
#
7d4df5cd |
|
25-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Use timeout from configuration file not only when sending and receiving, but also when establishing connection. MFC after: 1 week
|
#
643080b7 |
|
25-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Use role2str() when setting process title. MFC after: 1 week
|
#
9237aa3f |
|
22-Mar-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
After synchronization is complete we should make primary counters be equal to secondary counters: primary_localcnt = secondary_remotecnt primary_remotecnt = secondary_localcnt Previously it was done wrong and split-brain was observed after primary had synchronized up-to-date data from secondary. Approved by: pjd (mentor) MFC after: 1 week
|
#
b068d5aa |
|
22-Mar-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
For requests that are sent only to remote component use the error from remote. Approved by: pjd (mentor) MFC after: 1 week
|
#
cd72d521 |
|
22-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
White space cleanups. MFC after: 1 week
|
#
4d8dc3b8 |
|
21-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
When dropping privileges prefer capsicum over chroot+setgid+setuid. We can use capsicum for secondary worker processes and hastctl. When working as primary we drop privileges using chroot+setgid+setuid still as we need to send ioctl(2)s to ggate device, for which capsicum doesn't allow (yet). X-MFC after: capsicum is merged to stable/8
|
#
9446b453 |
|
21-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Initialize localcnt on first write. This fixes assertion when we create resource, set role to primary, do no writes, then sent it to secondary and accept connection from primary. MFC after: 1 week
|
#
0b626a28 |
|
21-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
In hast.conf we define the other node's address in 'remote' variable. This way we know how to connect to secondary node when we are primary. The same variable is used by the secondary node - it only accepts connections from the address stored in 'remote' variable. In cluster configurations it is common that each node has its individual IP address and there is one addtional shared IP address which is assigned to primary node. It seems it is possible that if the shared IP address is from the same network as the individual IP address it might be choosen by the kernel as a source address for connection with the secondary node. Such connection will be rejected by secondary, as it doesn't come from primary node individual IP. Add 'source' variable that allows to specify source IP address we want to bind to before connecting to the secondary node. MFC after: 1 week
|
#
8d7dcf14 |
|
17-Mar-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
For secondary, set 2 * HAST_KEEPALIVE seconds timeout for incoming connection so the worker will exit if it does not receive packets from the primary during this interval. Reported by: Christian Vogt <Christian.Vogt@haw-hamburg.de> Tested by: Christian Vogt <Christian.Vogt@haw-hamburg.de> Approved by: pjd (mentor) MFC after: 1 week
|
#
bc7a916a |
|
10-Mar-2011 |
Mikolaj Golub <trociny@FreeBSD.org> |
Make workers inherit debug level from the main process. Approved by: pjd (mentor) MFC after: 1 week
|
#
fa356f6c |
|
07-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
- Log size of data to synchronize in human readable form (using %N). - Log synchronization time (using %T). - Log synchronization speed in human readable form (using %N). MFC after: 2 weeks
|
#
8cd3d45a |
|
06-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Allow to compress on-the-wire data using two algorithms: - HOLE - it simply turns all-zero blocks into few bytes header; it is extremely fast, so it is turned on by default; it is mostly intended to speed up initial synchronization where we expect many zeros; - LZF - very fast algorithm by Marc Alexander Lehmann, which shows very decent compression ratio and has BSD license. MFC after: 2 weeks
|
#
1fee97b0 |
|
06-Mar-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Allow to checksum on-the-wire data using either CRC32 or SHA256. MFC after: 2 weeks
|
#
32ecf620 |
|
03-Feb-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Setup another socketpair between parent and child, so that primary sandboxed worker can ask the main privileged process to connect in worker's behalf and then we can migrate descriptor using this socketpair to worker. This is not really needed now, but will be needed once we start to use capsicum for sandboxing. MFC after: 1 week
|
#
21e7bc5e |
|
03-Feb-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Add missing locking after moving keepalive_send() to remote send thread in r214692. MFC after: 1 week
|
#
f4c96f94 |
|
03-Feb-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Let the caller log info about successful privilege drop. We don't want to log this in hastctl. MFC after: 1 week
|
#
9d70b24b |
|
02-Feb-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Allow to specify connection timeout by the caller. MFC after: 1 week
|
#
2ec483c5 |
|
31-Jan-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
- Use pjdlog for assertions and aborts as this will log assert/abort message to syslog if we run in background. - Asserts in proto.c that method we want to call is implemented and remove dummy methods from protocols implementation that are only there to abort the program with nice message. MFC after: 1 week
|
#
6d7967de |
|
28-Jan-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Drop privileges in worker processes. Accepting connections and handshaking in secondary is still done before dropping privileges. It should be implemented by only accepting connections in privileged main process and passing connection descriptors to the worker, but is not implemented yet. MFC after: 1 week
|
#
f463896e |
|
28-Jan-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Use newly added descriptors_assert() function to ensure only expected descriptors are open. MFC after: 1 week
|
#
da1783ea |
|
28-Jan-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Close all unneeded descriptors after fork(2). MFC after: 1 week
|
#
d64c0992 |
|
28-Jan-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Add comments to places where we treat errors as ciritical, but it is possible to handle them more gracefully. MFC after: 1 week
|
#
115f4e5c |
|
24-Jan-2011 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Don't open configuration file from worker process. Handle SIGHUP in the master process only and pass changes to the worker processes over control socket. This removes access to global namespace in preparation for capsicum sandboxing. MFC after: 2 weeks
|
#
fba1bf5a |
|
16-Dec-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
The 'ret' variable is of type ssize_t and we use proper format for it (%zd), so no (bogus) cast is needed. MFC after: 3 days
|
#
cd7b7ee5 |
|
16-Dec-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Improve problems logging. MFC after: 3 days
|
#
72089204 |
|
16-Dec-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Don't ignore errors from remote requests. MFC after: 3 days
|
#
d448536c |
|
14-Nov-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Move timeout.tv_sec initialization outside the loop - sigtimedwait(2) won't modify it. Submitted by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
1dd5a4bf |
|
14-Nov-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
1. Exit when we cannot create incoming connection. 2. Improve logging to inform which connection can't be created. Submitted by: [1] Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
448efa94 |
|
02-Nov-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Send packets to remote node only via the send thread to avoid possible races - in this case a keepalive packet was send from wrong thread which lead to connection dropping, because of corrupted packet. Fix it by sending keepalive packets directly from the send thread. As a bonus we now send keepalive packets only when connection is idle. Submitted by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
ce837469 |
|
24-Oct-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Before this change on first connect between primary and secondary we initialize all the data. This is huge waste of time and resources if there were no writes yet, as there is no real data to synchronize. Optimize this by sending "virgin" argument to secondary, which gives it a hint that synchronization is not needed. In the common case (where noth nodes are configured at the same time) instead of synchronizing everything, we don't synchronize at all. MFC after: 1 week
|
#
584a9bc3 |
|
24-Oct-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Plug memory leaks. Found with: valgrind MFC after: 3 days
|
#
a7d5f7eb |
|
19-Oct-2010 |
Jamie Gritton <jamie@FreeBSD.org> |
A new jail(8) with a configuration file, to replace the work currently done by /etc/rc.d/jail.
|
#
51c63dce |
|
08-Oct-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
We can't zero out ggio request, as we have some fields in there we initialize once during start-up. Reported by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
4e47b646 |
|
07-Oct-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Clear ggate structures before using them. We don't initialize all the field and there can be some garbage from the stack. MFC after: 1 week
|
#
783ee753 |
|
07-Oct-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Log error message when we fail to destroy ggate provider. MFC after: 3 days
|
#
4a88128b |
|
07-Oct-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Start the guard thread first, so we can handle signals from the very begining. Reported by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 1 week
|
#
b46198a5 |
|
07-Oct-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Don't close local component on exit as we can hang waiting on g_waitidle. I'm unable to reproduce the race described in comment anymore and also the comment is incorrect - localfd represents local component from configuration file, eg. /dev/da0 and not HAST provider. Reported by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 1 week
|
#
9dd5a6cb |
|
22-Sep-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Switch to sigprocmask(2) API also in the main process and secondary process. This way the primary process inherits signal mask from the main process, which fixes a race where signal is delivered to the primary process before configuring signal mask. Reported by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
8b70e6ae |
|
22-Sep-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Fix possible deadlock where worker process sends an event to the main process while the main process sends control message to the worker process, but worker process hasn't started control thread yet, because it waits for reply from the main process. The fix is to start the control thread before sending any events. Reported and fix suggested by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
e43e02f1 |
|
20-Sep-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Add __dead2 to functions that we know they are going to exit. MFC after: 3 days
|
#
852ac373 |
|
31-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Mask only those signals that we want to handle. Suggested by: jilles MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
5bdff860 |
|
30-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Because it is very hard to make fork(2) from threaded process safe (we are limited to async-signal safe functions in the child process), move all hooks execution to the main (non-threaded) process. Do it by maintaining connection (socketpair) between child and parent and sending events from the child to parent, so it can execute the hook. This is step in right direction for others reasons too. For example there is one less problem to drop privs in worker processes. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
6be3a25c |
|
30-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Use pjdlog_exit() before fork(). MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
5b41e644 |
|
29-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Execute hook when connection between the nodes is established or lost. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
2be8fd75 |
|
29-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Execute hook when split-brain is detected. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
6d0c801e |
|
29-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Use sigtimedwait(2) for signals handling in primary process. This fixes various races and eliminates use of pthread* API in signal handler. Pointed out by: kib With help from: jilles MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
ff6bb1f8 |
|
29-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
- Move functionality responsible for checking one connection to separate function to make code more readable. - Be sure not to reconnect too often in case of signal delivery, etc. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
ee087cdf |
|
29-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Disconnect after logging errors. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
ecc99c89 |
|
29-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Allow to run hooks from the main hastd process. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
b9cf0cf5 |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Correct when we log interrupted synchronization. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
eba09893 |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Check if no signals were delivered just before going to sleep. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
01125a93 |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Add hooks execution. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
0becad39 |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Allow to execute specified program on various HAST events. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
f7fe83f9 |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Implement keepalive mechanism inside HAST protocol so we can detect secondary node failures quickly for HAST resources that are rarely modified. Remove XXX from a comment now that the guard thread never sleeps infinitely. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
8f8c798c |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
- Remove redundant and incorrect 'old' word from debug message. - Log disconnects as warnings. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
e23d2d01 |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Don't increase number synchronized bytes in case of an error. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
53d9b386 |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Log that synchronization was interrupted in a proper place. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
55ce1e7c |
|
27-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
We have sync_start() function to start synchronization, introduce sync_stop() function to stop it. MFC after: 2 weeks Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com
|
#
0989854d |
|
05-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Implement configuration reload on SIGHUP. This includes: - Load added resources. - Stop and forget removed resources. - Update modified resources in least intrusive way, ie. don't touch /dev/hast/<name> unless path to local component or provider name were modified. Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com MFC after: 1 month
|
#
f377917c |
|
05-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Allow to use 'none' keywork as remote address in case second cluster node is not setup yet. MFC after: 1 month
|
#
a2ef0636 |
|
05-Aug-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Reset signal handlers after fork(). MFC after: 1 month
|
#
c9697b73 |
|
23-Jun-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
MFC r209263: r209175: Eliminate dead code. Found by: Coverity Prevent CID: 5158 r209177: Remove macros that are not really needed. The idea was to have them in case we grow more descriptors, but I'll reconsider readding them once we get there. Passing (a = b) expression to FD_ISSET() is bad idea, as FD_ISSET() evaluates its argument twice. Found by: Coverity Prevent CID: 5243 r209179: Plug memory leaks. Found by: Coverity Prevent CID: 7052, 7053, 7054, 7055 r209180: Plug memory leak. Found by: Coverity Prevent CID: 7051 r209181: Plug memory leak. Found by: Coverity Prevent CID: 7056 r209182: Plug memory leak. Found by: Coverity Prevent CID: 7057 r209183: Initialize gctl_seq for synchronization requests. Reported by: hiroshi@soupacific.com Analysed by: Mikolaj Golub <to.my.trociny@gmail.com> Tested by: hiroshi@soupacific.com, Mikolaj Golub <to.my.trociny@gmail.com> r209184: Fix typos. r209185: Correct various log messages. Submitted by: Mikolaj Golub <to.my.trociny@gmail.com> Note that without some of these changes hastd won't work on 8.x properly. Approved by: re (kensmith)
|
#
328e0f4b |
|
14-Jun-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Initialize gctl_seq for synchronization requests. Reported by: hiroshi@soupacific.com Analysed by: Mikolaj Golub <to.my.trociny@gmail.com> Tested by: hiroshi@soupacific.com, Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
b0dfbe5b |
|
14-Jun-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Plug memory leak. Found by: Coverity Prevent CID: 7056 MFC after: 3 days
|
#
badd3232 |
|
01-May-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
MFC r207070,r207343,r207345,r207347,r207348,r207371,r207372,r207390: r207070: Fix compilation with WITHOUT_CRYPT or WITHOUT_OPENSSL options. Reported by: Andrei V. Lavreniyuk <andy.lavr@reactor-xg.kiev.ua> r207343: Don't assume that "resource" property is in metadata. Reported by: Mikolaj Golub <to.my.trociny@gmail.com> r207345: Use WEXITSTATUS() to obtain real exit code. r207347: Mark temporary issues as such. r207348: Restart worker thread only if the problem was temporary. In case of persistent problem we don't want to loop forever. r207371: Fix a problem where hastd will stuck in recv(2) after sending request to secondary, which died between send(2) and recv(2). Do it by adding timeout to recv(2) for primary incoming and outgoing sockets and secondary outgoing socket. Reported by: Mikolaj Golub <to.my.trociny@gmail.com> Tested by: Mikolaj Golub <to.my.trociny@gmail.com> r207372: - Check if the worker process was killed by signal and restart it. - Improve logging. Pointed out by: Garrett Cooper <yanefbsd@gmail.com> r207390: Default connection timeout is way too long. To make it shorter we have to make socket non-blocking, connect() and if we get EINPROGRESS, we have to wait using select(). Very complex, but I know no other way to define connection timeout for a given socket. Reported by: hiroshi@soupacific.com
|
#
5571414c |
|
29-Apr-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Fix a problem where hastd will stuck in recv(2) after sending request to secondary, which died between send(2) and recv(2). Do it by adding timeout to recv(2) for primary incoming and outgoing sockets and secondary outgoing socket. Reported by: Mikolaj Golub <to.my.trociny@gmail.com> Tested by: Mikolaj Golub <to.my.trociny@gmail.com> MFC after: 3 days
|
#
5abfc9c1 |
|
28-Apr-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Mark temporary issues as such. MFC after: 3 days
|
#
c44c50cd |
|
18-Apr-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
MFC r204177,r205738,r206669,r206696,r206697: r204177: Changing proto_socketpair.c compilation and linking order revealed a problem - we should simply ignore proto_server() if address doesn't start with socketpair://, and not abort. r205738: Don't hold connection lock when doing reconnects as it makes I/Os wait for connection timeouts. Reported by: Kevin Day <toasty@dragondata.com> r206669: Increase ggate queue size to maximum value. HAST was not able to stand heavy random load. Reported by: Hiroyuki Yamagami r206696: Fix control socket leak when worker process exits. Submitted by: Mikolaj Golub <to.my.trociny@gmail.com> r206697: Fix log size calculation which caused message truncation. Submitted by: Mikolaj Golub <to.my.trociny@gmail.com>
|
#
2b98f840 |
|
18-Apr-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
MFC r204076,r204077,r204083,r205279: r204076: Please welcome HAST - Highly Avalable Storage. HAST allows to transparently store data on two physically separated machines connected over the TCP/IP network. HAST works in Primary-Secondary (Master-Backup, Master-Slave) configuration, which means that only one of the cluster nodes can be active at any given time. Only Primary node is able to handle I/O requests to HAST-managed devices. Currently HAST is limited to two cluster nodes in total. HAST operates on block level - it provides disk-like devices in /dev/hast/ directory for use by file systems and/or applications. Working on block level makes it transparent for file systems and applications. There in no difference between using HAST-provided device and raw disk, partition, etc. All of them are just regular GEOM providers in FreeBSD. For more information please consult hastd(8), hastctl(8) and hast.conf(5) manual pages, as well as http://wiki.FreeBSD.org/HAST. Sponsored by: FreeBSD Foundation Sponsored by: OMCnet Internet Service GmbH Sponsored by: TransIP BV r204077: Remove some lines left over by accident. r204083: Add missing KEYWORD line. Pointed out by: dougb r205279 sys: Simplify loops.
|
#
20b77db9 |
|
15-Apr-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Increase ggate queue size to maximum value. HAST was not able to stand heavy random load. Reported by: Hiroyuki Yamagami MFC after: 3 days
|
#
0d9014f3 |
|
27-Mar-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Don't hold connection lock when doing reconnects as it makes I/Os wait for connection timeouts. Reported by: Kevin Day <toasty@dragondata.com>
|
#
fe0506d7 |
|
09-Mar-2010 |
Marcel Moolenaar <marcel@FreeBSD.org> |
Create the altix project branch. The altix project will add support for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting is a two-module system, consisting of a base compute module and a CPU expansion module. SGI's NUMAFlex architecture can be an excellent platform to test CPU affinity and NUMA-aware features in FreeBSD.
|
#
32115b10 |
|
18-Feb-2010 |
Pawel Jakub Dawidek <pjd@FreeBSD.org> |
Please welcome HAST - Highly Avalable Storage. HAST allows to transparently store data on two physically separated machines connected over the TCP/IP network. HAST works in Primary-Secondary (Master-Backup, Master-Slave) configuration, which means that only one of the cluster nodes can be active at any given time. Only Primary node is able to handle I/O requests to HAST-managed devices. Currently HAST is limited to two cluster nodes in total. HAST operates on block level - it provides disk-like devices in /dev/hast/ directory for use by file systems and/or applications. Working on block level makes it transparent for file systems and applications. There in no difference between using HAST-provided device and raw disk, partition, etc. All of them are just regular GEOM providers in FreeBSD. For more information please consult hastd(8), hastctl(8) and hast.conf(5) manual pages, as well as http://wiki.FreeBSD.org/HAST. Sponsored by: FreeBSD Foundation Sponsored by: OMCnet Internet Service GmbH Sponsored by: TransIP BV
|