sbi_ipi_event_create() disallows registering an IPI event with a NULL
.process callback, so the function pointer will never be NULL here.
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Xiang W <wxjstz@126.com>
An IPI sent to the local hart can be processed directly instead of
triggering the IPI device. This is more efficient, and it avoids a
deadlock when the .sync callback is defined. Since interrupts are
disabled while handling an ecall, the IPI would not get delivered
until the next mret, but sbi_ipi_sync() is called before then.
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Xiang W <wxjstz@126.com>
Currently, failures in sbi_ipi_send() are silently ignored, which makes
them difficult to debug. Instead, abort sending the IPI and pass back
the error, but still synchronize any IPIs already sent.
Signed-off-by: Samuel Holland <samuel.holland@sifive.com>
Signed-off-by: Anup Patel <anup@brainfault.org>
Multiple harts may try to send IPI to a particular target hart A
in which case the send_ipi() should be called only when the old
value of the hart A ipi_type is zero.
Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Currently, there are no barriers before or after the ipi_clear()
device callback which forces ipi_clear() device callback to always
use non-relaxed MMIO writes.
Instead of above, we use wmb() in after the ipi_clear() device
callback which pairs with the wmb() done before the ipi_send()
device callback. This also allows device ipi_clear() callback
to use relaxed MMIO writes.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
Reported-by: Bo Gan <ganboing@gmail.com>
Currently, we have a smp_wmb() between atomic_raw_set_bit() and
ipi_send() device callback whereas the MMIO writes done by the
device ipi_send() callback will also include a barrier.
We can avoid unnecessary/redundant barriers described above by
allowing relaxed MMIO writes in device ipi_send() callback. To
achieve this, we simply use wmb() instead of smp_wmb() before
calling device ipi_send().
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
Reported-by: Bo Gan <ganboing@gmail.com>
Let us prefer hartindex over hartid in IPI framework which in-turn
forces IPI users to also prefer hartindex.
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
Currently, the sbi_hartmask is indexed by hartid which puts a
limit on hartid to be less than SBI_HARTMASK_MAX_BITS.
We extend the sbi_hartmask implementation to use hartindex and
support updating sbi_hartmask using hartid. This removes the
limit on hartid and existing code works largely unmodified.
Signed-off-by: Xiang W <wxjstz@126.com>
Signed-off-by: Anup Patel <apatel@ventanamicro.com>
The previous definition had the assumption that the machine word length
is equal to the word length of LONG. Remove this assumption and add a
static check to prevent errors in subsequent modifications.
Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
We have redundant semicolon at quite a few places so let's remove it.
Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
When tlb_fifo is full, it will wait and affect the ipi update to
other harts. This patch is optimized.
Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Tested-by: Anup Patel <anup@brainfault.org>
The original sbi_ipi will be processed by hart by hart, after optimization,
send ipi first and finally wait together.
Signed-off-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
Tested-by: Anup Patel <anup@brainfault.org>
Since commit 50d4fde1c5 ("lib: Remove redundant sbi_platform_ipi_clear()
calls"), the IPI sent from the boot hart in wake_coldboot_harts() is not
cleared in the secondary harts until they reach sbi_ipi_init(). However,
sbi_hsm_init() and sbi_hsm_hart_wait() are called earlier, so a secondary
hart might enter sbi_hsm_hart_wait() with an already pending IPI.
sbi_hsm_hart_wait() makes sure the hart leaves the loop only when it is
actually ready, so a pending unrelated IPI should not cause safety issues.
However, it might be inefficient on certain hardware, because it prevents
"wfi" from stalling the hart even if the hardware supports this, making the
hart needlessly spin in a "busy-wait" loop.
This behaviour can be observed, for example, in a QEMU VM (QEMU 7.2.0) with
"-machine virt" running a Linux guest. Inserting delays in
sbi_hsm_hart_start() allows reproducing the issue more reliably.
The comment in wait_for_coldboot() suggests that the initial IPI is needed
in the warm resume path, so let us clear it before init_warm_startup()
only.
To do this, sbi_ipi_raw_clear() was created similar to sbi_ipi_raw_send().
Signed-off-by: Evgenii Shatokhin <e.shatokhin@yadro.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
C language standard uses true/false for the boolean type.
Let's switch to that for better language compatibility.
Signed-off-by: Bin Meng <bmeng@tinylab.org>
Reviewed-by: Anup Patel <anup@brainfault.org>
Reviewed-by: Samuel Holland <samuel@sholland.org>
Tested-by: Samuel Holland <samuel@sholland.org>
If the ecall SBI_EXT_HSM_HART_START is called it might try to wake the
secondary hart using sbi_ipi_raw_send() to send an IPI to the hart.
This can fail if there is no IPI device but no error is returned from
sbi_ipi_raw_send() so the ecall returns as if the action completed and
the caller continues without noticing (in the case of Linux it just hangs
waiting for the secondary hart to become active)
Fix this by changing sbi_ipi_raw_send() to return and error, and if an
error is returned, then return it via SBI_EXT_HSM_HART_START call.
Signed-off-by: Ben Dooks <ben.dooks@sifive.com>
Reviewed-by: Anup Patel <anup@brainfault.org>
RISC-V SBI v0.3 specification defines a set of firmware events that can
provide additional information about the current firmware context. All
of the firmware event monitoring are enabled now. The firmware
events must be defined as raw perf event with MSB set as specified in the
specification.
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Signed-off-by: Atish Patra <atish.patra@wdc.com>
The parameter owner of function sbi_scratch_alloc_offset() is never used.
The scratch memory is small. We should not use it for debug information in
future. Hence eliminate the parameter.
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Xiang W <wxjstz@126.com>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Instead of having ipi_send() and ipi_clear() callbacks in
platform operations, it will be much simpler for ipi driver
to directly register these operations as a device to sbi_ipi
implementation.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
The sbi_platform_ipi_clear() called from wait_for_coldboot() and
sbi_hsm_hart_wait() is redundant because IPI will be automatically
cleared by sbi_platform_ipi_init() called from sbi_ipi_init().
Further, wait_for_coldboot() is common for warm startup and warm
resume path so the sbi_platform_ipi_clear() called in warm resume
path cause resuming HART to miss an IPI injected other HART to
wakeup the HART.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
A hart can take interrupt in the new HSM states introduced by the
SBI HSM suspend function (such as SUSPENDED state) so we rename
sbi_hsm_hart_started_mask() to something more generic such as
sbi_hsm_hart_interruptible_mask().
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
The sbi_hsm_hart_started_mask() API should take one more parameter
to allow caller specify domain under which started_mask is being
generated. Further, the sbi_hsm_hart_started_mask() depends on
sbi_hsm_hart_get_state() which also should return HART state under
specified domain.
This patch updates both sbi_hsm_hart_started_mask() and
sbi_hsm_hart_get_state() as-per above.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
The scratch parameter in sbi_hsm_hart_started_mask() API is now
redundant hence removing it.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
The sbi_hartid_to_scratch() API can fail for non-existent HARTs so
all uses of sbi_hartid_to_scratch() API should check return value.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
This patch simplify HART id to scratch macro as follows:
1. Remove current "scratch" pointer argument because now we
use HART id to scratch table
2. Rename sbi_hart_id_to_scratch() to sbi_hartid_to_scratch()
to have macro name consistent with the name of callback
in struct sbi_scratch
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
The sbi_current_hartid() being a regular function is quite
expensive because for callers it is a function call instead
of a direct CSR read. This patch converts sbi_current_hartid()
into a macro in riscv_asm.h.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
This patch replaces use of sbi_hart_available_mask() API with
sbi_hsm_hart_started_mask() API.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
This patch adds support for hart hotplug in OpenSBI using a generic WFI
based approach. Hart hotplug can be achieved via SBI HSM extension which
allows supervisor mode software to start or stop any harts anytime.
Any platform wishes to implement platform specific hart hotplug must
implement both hart_start and hart_stop in addition to enable platform
feature SBI_PLATFORM_HAS_HART_HOTPLUG.
Signed-off-by: Atish Patra <atish.patra@wdc.com>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
We don't need to separately call sbi_ipi_send() for current HART
in sbi_ipi_send_many(). Instead, we can simplify the for-loop in
sbi_ipi_send_many() and call sbi_ipi_send() for all HARTs in the
for-loop itself.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
As per the latest SBI specification, all online harts should receive
IPI if hbase is set to -1.
Set the target mask to all available hart mask if hbase is -1.
Signed-off-by: Atish Patra <atish.patra@wdc.com>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
Currently, the remote TLB management is initialized via IPI init
which is counter intuitive. This patch initializes remote TLB
management directly from init_coldboot() and init_warmboot()
after IPI init is done.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
This patch drops _fifo from the name of various sbi_tlb_fifo_xyz()
functions because all these functions deal with remote TLB managment
and FIFO is the per-HART data structure used internally by remote
TLB implementation.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
This patch introduces sbi_ipi_event_create/destroy() APIs and
struct sbi_ipi_event_ops for creating/destroying IPI events
at runtime based of event operations.
This new APIs will help platform code and utils code to create
custom IPI events which are not part of generic OpenSBI library.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
Instead of directly calling sbi_ipi_send_many(), we introduce
sbi_ipi_send_halt() for halting a set of HARTs.
This way in future we can assign any IPI event number for HART
halting within sbi_ipi.c only.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
Instead of directly calling sbi_ipi_send_many(), we introduce
sbi_ipi_send_smode() for injecting S-mode software interrupts.
This way in future we can assign any IPI event number for S-mode
IPIs within sbi_ipi.c only.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
The struct sbi_ipi_data is only used in sbi_ipi.c so move it
to sbi_ipi.c from sbi_ipi.h.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
Instead of doing sbi_hang() we should do sbi_exit() upon
halt IPI.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
We add sbi_ipi_exit() API for exit path handling in sbi_exit()
implementation.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
This patch adds new IPI extension which replaces ipi related
v0.1 extensions. This also adds a new API for ipi sending as trap
handling is not necessary in v0.2 SBI IPI related extensions.
It also modifies the IPI sending code which now accepts hart mask as a value
instead of S-mode virtual address. Thus, the caller should set it to exact hart
mask value everytime.
Signed-off-by: Atish Patra <atish.patra@wdc.com>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
We just need to distinguish only between FENCE and non FENCE related
IPIs as all of the fence related requests are handled via fifo now.
Remove the unnecessary IPI types related to individual fence types.
Signed-off-by: Atish Patra <atish.patra@wdc.com>
Reviewed-by: Anup Patel <anup.patel@wdc.com>
The unpriv APIs can be useful to external firmware and out-of-tree
platform support code.
This patch adds "sbi_" prefix to unpriv load/store APIs and rename
struct riscv_unpriv to struct sbi_trap_info everywhere. We also
place struct sbi_trap_info in sbi/sbi_trap.h so that we can use
it for sbi_trap_redirect() as well.
Overall, this patch will make naming of unpriv APIs consistent
with other OpenSBI APIs.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Reviewed-by: Atish Patra <atish.patra@wdc.com>
Linux kernel expects tlb flush SBI call to be completely synchronous i.e.
the SBI call should only return once corresponding *fence* instruction is
executed.
OpenSBI manages the outstanding TLB flush requests by keeping them in a
per hart based fifo. However, there are few corner cases that may lead to
race conditions while updating the fifo.
Currently, the caller hart waits for IPI acknowledgement via clint
address which is not a very good method as synchronization on MMIO may not
be supported in every platform. Moreover, the waiter doesn't have any way of
identifying if the IPI is received for specific tlb flush request or any
other IPI. This may lead to unpredictable behavior in supervisor/user space.
Fix this by waiting on individual fifo entries rather than MMIO address.
Currently, a relaxed loop is being used because wfi again involves MMIO write
which would be slower compared to relaxed loop. To avoid deadlock, fifo
is processed every time a hart loops for fifo enqueue or fifo sync to consume
the tlb flush requests sent by other harts.
Signed-off-by: Anup Patel <anup.patel@wdc.com>
Signed-off-by: Atish Patra <atish.patra@wdc.com>
All string functions are part of libsbi. It makes more sense
to rename them to sbi_string.x as the libsbi can be linked
with external libraries that can have similar implementation.
Signed-off-by: Atish Patra <atish.patra@wdc.com>
Acked-by: Anup Patel <anup.patel@wdc.com>
libsbi needs some of the custom libc functions. It should be directly
included in libsbi instead of platform specific libraries.
Signed-off-by: Atish Patra <atish.patra@wdc.com>
Acked-by: Anup Patel <anup.patel@wdc.com>
