Star64_linux/arch/x86/kernel
Borislav Petkov (AMD) ccce12ecf2 x86/barrier: Do not serialize MSR accesses on AMD
commit 04c3024560d3a14acd18d0a51a1d0a89d29b7eb5 upstream.

AMD does not have the requirement for a synchronization barrier when
acccessing a certain group of MSRs. Do not incur that unnecessary
penalty there.

There will be a CPUID bit which explicitly states that a MFENCE is not
needed. Once that bit is added to the APM, this will be extended with
it.

While at it, move to processor.h to avoid include hell. Untangling that
file properly is a matter for another day.

Some notes on the performance aspect of why this is relevant, courtesy
of Kishon VijayAbraham <Kishon.VijayAbraham@amd.com>:

On a AMD Zen4 system with 96 cores, a modified ipi-bench[1] on a VM
shows x2AVIC IPI rate is 3% to 4% lower than AVIC IPI rate. The
ipi-bench is modified so that the IPIs are sent between two vCPUs in the
same CCX. This also requires to pin the vCPU to a physical core to
prevent any latencies. This simulates the use case of pinning vCPUs to
the thread of a single CCX to avoid interrupt IPI latency.

In order to avoid run-to-run variance (for both x2AVIC and AVIC), the
below configurations are done:

  1) Disable Power States in BIOS (to prevent the system from going to
     lower power state)

  2) Run the system at fixed frequency 2500MHz (to prevent the system
     from increasing the frequency when the load is more)

With the above configuration:

*) Performance measured using ipi-bench for AVIC:
  Average Latency:  1124.98ns [Time to send IPI from one vCPU to another vCPU]

  Cumulative throughput: 42.6759M/s [Total number of IPIs sent in a second from
  				     48 vCPUs simultaneously]

*) Performance measured using ipi-bench for x2AVIC:
  Average Latency:  1172.42ns [Time to send IPI from one vCPU to another vCPU]

  Cumulative throughput: 40.9432M/s [Total number of IPIs sent in a second from
  				     48 vCPUs simultaneously]

From above, x2AVIC latency is ~4% more than AVIC. However, the expectation is
x2AVIC performance to be better or equivalent to AVIC. Upon analyzing
the perf captures, it is observed significant time is spent in
weak_wrmsr_fence() invoked by x2apic_send_IPI().

With the fix to skip weak_wrmsr_fence()

*) Performance measured using ipi-bench for x2AVIC:
  Average Latency:  1117.44ns [Time to send IPI from one vCPU to another vCPU]

  Cumulative throughput: 42.9608M/s [Total number of IPIs sent in a second from
  				     48 vCPUs simultaneously]

Comparing the performance of x2AVIC with and without the fix, it can be seen
the performance improves by ~4%.

Performance captured using an unmodified ipi-bench using the 'mesh-ipi' option
with and without weak_wrmsr_fence() on a Zen4 system also showed significant
performance improvement without weak_wrmsr_fence(). The 'mesh-ipi' option ignores
CCX or CCD and just picks random vCPU.

  Average throughput (10 iterations) with weak_wrmsr_fence(),
        Cumulative throughput: 4933374 IPI/s

  Average throughput (10 iterations) without weak_wrmsr_fence(),
        Cumulative throughput: 6355156 IPI/s

[1] https://github.com/bytedance/kvm-utils/tree/master/microbenchmark/ipi-bench

Signed-off-by: Borislav Petkov (AMD) <bp@alien8.de>
Link: https://lore.kernel.org/r/20230622095212.20940-1-bp@alien8.de
Signed-off-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2024-02-23 09:25:27 +01:00
..
acpi x86/i8259: Skip probing when ACPI/MADT advertises PCAT compatibility 2023-10-27 20:36:49 +02:00
apic x86/apic/msi: Fix misconfigured non-maskable MSI quirk 2023-11-28 17:19:56 +00:00
cpu x86/barrier: Do not serialize MSR accesses on AMD 2024-02-23 09:25:27 +01:00
fpu x86/fpu: Stop relying on userspace for info to fault in xsave buffer 2024-02-23 09:25:10 +01:00
kprobes x86/kprobes: fix incorrect return address calculation in kprobe_emulate_call_indirect 2024-01-10 17:17:00 +01:00
.gitignore
alternative.c x86/alternatives: Disable interrupts and sync when optimizing NOPs in place 2024-01-01 12:42:47 +00:00
amd_gart_64.c x86/mm: Remove P*D_PAGE_MASK and P*D_PAGE_SIZE macros 2022-12-15 10:37:27 -08:00
amd_nb.c x86/amd_nb: Use Family 19h Models 60h-7Fh Function 4 IDs 2023-11-20 11:59:38 +01:00
aperture_64.c
apm_32.c x86/APM: drop the duplicate APM_MINOR_DEV macro 2023-07-30 14:00:32 +02:00
asm-offsets.c x86/smpboot: Remove initial_stack on 64-bit 2023-03-21 13:35:53 +01:00
asm-offsets_32.c
asm-offsets_64.c
audit_64.c x86/audit: Fix -Wmissing-variable-declarations warning for ia32_xyz_class 2023-08-30 10:11:16 +02:00
bootflag.c
callthunks.c x86,static_call: Fix static-call vs return-thunk 2023-09-22 18:58:24 +02:00
cet.c x86/ibt: Convert IBT selftest to asm 2023-08-17 17:07:09 +02:00
cfi.c
check.c
cpuid.c x86/cpuid: make cpuid_class a static const structure 2023-08-05 08:31:41 +02:00
crash.c ARM: 2023-09-07 13:52:20 -07:00
crash_core_32.c
crash_core_64.c
crash_dump_32.c
crash_dump_64.c
devicetree.c x86/apic: Make some APIC init functions bool 2023-08-09 11:58:20 -07:00
doublefault_32.c x86: Avoid missing-prototype warnings for doublefault code 2023-05-18 11:56:18 -07:00
dumpstack.c x86/show_trace_log_lvl: Ensure stack pointer is aligned, again 2023-05-16 06:31:04 -07:00
dumpstack_32.c
dumpstack_64.c
e820.c x86/setup: Move duplicate boot_cpu_data definition out of the ifdeffery 2023-01-11 12:45:16 +01:00
early-quirks.c
early_printk.c
ebda.c
eisa.c
espfix_64.c
ftrace.c x86/ftrace: Remove unsued extern declaration ftrace_regs_caller_ret() 2023-07-10 21:38:13 -04:00
ftrace_32.S x86/ftrace: Enable HAVE_FUNCTION_GRAPH_RETVAL 2023-06-20 18:38:38 -04:00
ftrace_64.S x86/ftrace: Enable HAVE_FUNCTION_GRAPH_RETVAL 2023-06-20 18:38:38 -04:00
head32.c x86: Add dummy prototype for mk_early_pgtbl_32() 2023-05-18 11:56:16 -07:00
head64.c x86/boot: Fix incorrect startup_gdt_descr.size 2023-11-20 11:58:53 +01:00
head_32.S x86/smpboot: Restrict soft_restart_cpu() to SEV 2023-05-15 13:44:50 +02:00
head_64.S x86/smpboot/64: Handle X2APIC BIOS inconsistency gracefully 2024-01-01 12:42:47 +00:00
hpet.c rtc: Add support for configuring the UIP timeout for RTC reads 2024-01-31 16:18:56 -08:00
hw_breakpoint.c x86/amd: Cache debug register values in percpu variables 2023-01-31 20:09:26 +01:00
i8237.c
i8253.c
i8259.c x86/i8259: Skip probing when ACPI/MADT advertises PCAT compatibility 2023-10-27 20:36:49 +02:00
ibt_selftest.S x86/ibt: Convert IBT selftest to asm 2023-08-17 17:07:09 +02:00
idt.c x86/entry: Convert INT 0x80 emulation to IDTENTRY 2023-12-13 18:45:02 +01:00
io_delay.c
ioport.c
irq.c x86/apic: Nuke ack_APIC_irq() 2023-08-09 11:58:34 -07:00
irq_32.c
irq_64.c
irq_work.c x86/apic: Wrap IPI calls into helper functions 2023-08-09 12:00:55 -07:00
irqflags.S
irqinit.c x86/i8259: Mark legacy PIC interrupts with IRQ_LEVEL 2023-01-16 17:24:56 +01:00
itmt.c x86/sched/itmt: Give all SMT siblings of a core the same priority 2023-05-08 10:58:38 +02:00
jailhouse.c x86/apic: Remove the pointless APIC version check 2023-08-09 11:58:19 -07:00
jump_label.c
kdebugfs.c
kexec-bzimage64.c docs: move x86 documentation into Documentation/arch/ 2023-03-30 12:58:51 -06:00
kgdb.c x86/kgdb: Fix a kerneldoc warning when build with W=1 2023-09-24 11:00:13 +02:00
ksysfs.c
kvm.c * Rework apic callbacks, getting rid of unnecessary ones and 2023-08-30 10:44:46 -07:00
kvmclock.c x86/kvm: Do not try to disable kvmclock if it was not enabled 2024-01-25 15:35:48 -08:00
ldt.c x86: allow get_locked_pte() to fail 2023-06-19 16:19:10 -07:00
machine_kexec_32.c
machine_kexec_64.c x86/kexec: remove unnecessary arch_kexec_kernel_image_load() 2023-04-08 13:45:38 -07:00
Makefile Add x86 shadow stack support 2023-08-31 12:20:12 -07:00
mmconf-fam10h_64.c
module.c x86/alternative: Rename apply_ibt_endbr() 2023-07-10 09:52:23 +02:00
mpparse.c x86/apic: Sanitize APIC address setup 2023-08-09 11:58:20 -07:00
msr.c x86/MSR: make msr_class a static const structure 2023-08-05 08:31:42 +02:00
nmi.c x86/nmi: Fix out-of-order NMI nesting checks & false positive warning 2023-11-20 11:58:53 +01:00
nmi_selftest.c x86/apic: Wrap IPI calls into helper functions 2023-08-09 12:00:55 -07:00
paravirt-spinlocks.c
paravirt.c x86/xen: move paravirt lazy code 2023-09-19 07:04:49 +02:00
pci-dma.c x86: always initialize xen-swiotlb when xen-pcifront is enabling 2023-07-31 17:54:27 +02:00
pcspeaker.c
perf_regs.c
platform-quirks.c x86/quirks: Include linux/pnp.h for arch_pnpbios_disabled() 2023-05-18 11:56:18 -07:00
pmem.c
probe_roms.c
process.c x86/shstk: Remove useless clone error handling 2023-09-19 09:18:34 -07:00
process.h
process_32.c x86/resctl: fix scheduler confusion with 'current' 2023-03-08 11:48:11 -08:00
process_64.c x86/shstk: Add ARCH_SHSTK_STATUS 2023-08-02 15:01:51 -07:00
ptrace.c x86: Add PTRACE interface for shadow stack 2023-08-02 15:01:51 -07:00
pvclock.c locking/atomic: treewide: use raw_atomic*_<op>() 2023-06-05 09:57:20 +02:00
quirks.c
reboot.c x86/reboot: Expose VMCS crash hooks if and only if KVM_{INTEL,AMD} is enabled 2023-08-03 15:37:14 -07:00
reboot_fixups_32.c
relocate_kernel_32.S
relocate_kernel_64.S x86,objtool: Split UNWIND_HINT_EMPTY in two 2023-03-23 23:18:58 +01:00
resource.c x86/PCI: Tidy E820 removal messages 2022-12-10 10:33:11 -06:00
rethook.c
rtc.c rtc: Extend timeout for waiting for UIP to clear to 1s 2024-01-31 16:18:56 -08:00
setup.c x86/mm, kexec, ima: Use memblock_free_late() from ima_free_kexec_buffer() 2023-09-18 09:24:15 +02:00
setup_percpu.c x86/apic/32: Remove x86_cpu_to_logical_apicid 2023-08-09 11:58:23 -07:00
sev-shared.c Take care of a race between when the #VC exception is raised and when 2023-10-19 18:12:08 -07:00
sev.c x86/sev: Fix kernel crash due to late update to read-only ghcb_version 2023-12-13 18:45:30 +01:00
sev_verify_cbit.S
shstk.c x86/shstk: Add warning for shadow stack double unmap 2023-09-19 09:18:34 -07:00
signal.c x86/shstk: Handle signals for shadow stack 2023-08-02 15:01:50 -07:00
signal_32.c x86/shstk: Add user control-protection fault handler 2023-08-02 15:01:50 -07:00
signal_64.c x86/shstk: Delay signal entry SSP write until after user accesses 2023-11-28 17:19:55 +00:00
smp.c Revert "x86/smp: Put CPUs into INIT on shutdown if possible" 2023-10-15 12:02:02 -07:00
smpboot.c Revert "x86/smp: Put CPUs into INIT on shutdown if possible" 2023-10-15 12:02:02 -07:00
stacktrace.c
static_call.c x86/static_call: Fix __static_call_fixup() 2023-08-17 13:24:09 +02:00
step.c
sys_ia32.c
sys_x86_64.c x86/mm: Introduce MAP_ABOVE4G 2023-07-11 14:12:19 -07:00
tboot.c
time.c
tls.c x86/gsseg: Move load_gs_index() to its own new header file 2023-01-12 13:06:36 +01:00
tls.h
topology.c cpu-hotplug: Provide prototypes for arch CPU registration 2023-10-11 14:27:37 +02:00
trace.c
trace_clock.c
tracepoint.c
traps.c Add x86 shadow stack support 2023-08-31 12:20:12 -07:00
tsc.c x86/tsc: Extend watchdog check exemption to 4-Sockets platform 2023-07-14 15:17:09 -07:00
tsc_msr.c
tsc_sync.c x86/tsc: Defer marking TSC unstable to a worker 2023-10-27 20:36:57 +02:00
umip.c
unwind_frame.c
unwind_guess.c
unwind_orc.c objtool changes for v6.5: 2023-06-27 15:05:41 -07:00
uprobes.c uprobes/x86: Allow to probe a NOP instruction with 0x66 prefix 2022-12-05 11:55:18 +01:00
verify_cpu.S
vm86_32.c
vmlinux.lds.S x86/build: Fix linker fill bytes quirk/incompatibility for ld.lld 2023-09-06 23:49:12 +02:00
vsmp_64.c x86/apic: Get rid of hard_smp_processor_id() 2023-08-09 11:58:17 -07:00
x86_init.c - Fix a race window where load_unaligned_zeropad() could cause 2023-06-26 16:32:47 -07:00