svn commit: r327706 - in head/sys: conf contrib/zstd/lib/freebsd

Tue Jan 9 09:47:49 UTC 2018

On Mon, 8 Jan 2018 20:14:16 +0000 (UTC)
Conrad Meyer <cem at FreeBSD.org> wrote:

> Author: cem
> Date: Mon Jan  8 20:14:16 2018
> New Revision: 327706
> URL: https://svnweb.freebsd.org/changeset/base/327706
> 
> Log:
>   Integrate zstd into the kernel
>   
>   Mock userspace headers and include mocked headers first in compilation
>   command to inject kernel headers and override e.g., malloc(3) with
>   malloc(9).
>   
>   Submitted by:	allanjude
>   Reviewed by:	imp (earlier version), bapt (earlier version)
>   Differential Revision:	https://reviews.freebsd.org/D10407
> 
> Added:
>   head/sys/contrib/zstd/lib/freebsd/
>   head/sys/contrib/zstd/lib/freebsd/stddef.h   (contents, props changed)
>   head/sys/contrib/zstd/lib/freebsd/stdint.h   (contents, props changed)
>   head/sys/contrib/zstd/lib/freebsd/stdio.h   (contents, props changed)
>   head/sys/contrib/zstd/lib/freebsd/stdlib.h   (contents, props changed)
>   head/sys/contrib/zstd/lib/freebsd/string.h   (contents, props changed)
>   head/sys/contrib/zstd/lib/freebsd/zstd_kfreebsd.h   (contents, props
> changed) head/sys/contrib/zstd/lib/freebsd/zstd_kmalloc.c   (contents, props
> changed) Modified:
>   head/sys/conf/files
>   head/sys/conf/kern.pre.mk
> 
> Modified: head/sys/conf/files
> ==============================================================================
> --- head/sys/conf/files	Mon Jan  8 18:47:35 2018	(r327705)
> +++ head/sys/conf/files	Mon Jan  8 20:14:16 2018	(r327706)
> @@ -626,6 +626,23 @@ contrib/ngatm/netnatm/sig/sig_unimsgcpy.c optional nga
>  	compile-with "${NORMAL_C} -I$S/contrib/ngatm"
>  contrib/ngatm/netnatm/sig/sig_verify.c optional ngatm_uni \
>  	compile-with "${NORMAL_C} -I$S/contrib/ngatm"
> +# Zstd
> +contrib/zstd/lib/freebsd/zstd_kmalloc.c		standard compile-with
> ${ZSTD_C} +contrib/zstd/lib/common/zstd_common.c		standard
> compile-with ${ZSTD_C} +contrib/zstd/lib/common/fse_decompress.c
> standard compile-with ${ZSTD_C}
> +contrib/zstd/lib/common/entropy_common.c	standard compile-with
> ${ZSTD_C} +contrib/zstd/lib/common/error_private.c		standard
> compile-with ${ZSTD_C} +contrib/zstd/lib/common/xxhash.c
> standard compile-with ${ZSTD_C}
> +contrib/zstd/lib/compress/zstd_compress.c	standard compile-with
> ${ZSTD_C} +contrib/zstd/lib/compress/fse_compress.c	standard
> compile-with ${ZSTD_C} +contrib/zstd/lib/compress/huf_compress.c
> standard compile-with ${ZSTD_C}
> +contrib/zstd/lib/compress/zstd_double_fast.c	standard compile-with
> ${ZSTD_C} +contrib/zstd/lib/compress/zstd_fast.c		standard
> compile-with ${ZSTD_C} +contrib/zstd/lib/compress/zstd_lazy.c
> standard compile-with ${ZSTD_C}
> +contrib/zstd/lib/compress/zstd_ldm.c		standard compile-with
> ${ZSTD_C} +contrib/zstd/lib/compress/zstd_opt.c		standard
> compile-with ${ZSTD_C} +contrib/zstd/lib/decompress/zstd_decompress.c
> standard compile-with ${ZSTD_C}
> +contrib/zstd/lib/decompress/huf_decompress.c	standard compile-with
> ${ZSTD_C} crypto/blowfish/bf_ecb.c	optional ipsec | ipsec_support
> crypto/blowfish/bf_skey.c	optional crypto | ipsec | ipsec_support
> crypto/camellia/camellia.c	optional crypto | ipsec | ipsec_support
> 
> Modified: head/sys/conf/kern.pre.mk
> ==============================================================================
> --- head/sys/conf/kern.pre.mk	Mon Jan  8 18:47:35 2018	(r327705)
> +++ head/sys/conf/kern.pre.mk	Mon Jan  8 20:14:16 2018	(r327706)
> @@ -132,6 +132,9 @@ NORMAL_FW= uudecode -o ${.TARGET} ${.ALLSRC}
>  NORMAL_FWO= ${LD} -b binary --no-warn-mismatch -d -warn-common -r \
>  	-m ${LD_EMULATION} -o ${.TARGET} ${.ALLSRC:M*.fw}
>  
> +# for ZSTD in the kernel (include zstd/lib/freebsd before other CFLAGS)
> +ZSTD_C= ${CC} -c -DZSTD_HEAPMODE=1 -I$S/contrib/zstd/lib/freebsd ${CFLAGS}
> -I$S/contrib/zstd/lib -I$S/contrib/zstd/lib/common ${WERROR}
> -Wno-missing-prototypes ${PROF} ${.IMPSRC} + # Common for dtrace / zfs
>  CDDL_CFLAGS=	-DFREEBSD_NAMECACHE -nostdinc
> -I$S/cddl/compat/opensolaris -I$S/cddl/contrib/opensolaris/uts/common -I$S
> -I$S/cddl/contrib/opensolaris/common ${CFLAGS} -Wno-unknown-pragmas
> -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual
> -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized
> -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -Wno-unknown-pragmas
> CDDL_CFLAGS+=	-include $S/cddl/compat/opensolaris/sys/debug_compat.h
> 
> Added: head/sys/contrib/zstd/lib/freebsd/stddef.h
> ==============================================================================
> --- /dev/null	00:00:00 1970	(empty, because file is newly added)
> +++ head/sys/contrib/zstd/lib/freebsd/stddef.h	Mon Jan  8 20:14:16
> 2018	(r327706) @@ -0,0 +1,3 @@
> +/* This file is in the public domain */
> +/* $FreeBSD$ */
> +#include "zstd_kfreebsd.h"
> 
> Added: head/sys/contrib/zstd/lib/freebsd/stdint.h
> ==============================================================================
> --- /dev/null	00:00:00 1970	(empty, because file is newly added)
> +++ head/sys/contrib/zstd/lib/freebsd/stdint.h	Mon Jan  8 20:14:16
> 2018	(r327706) @@ -0,0 +1,3 @@
> +/* This file is in the public domain */
> +/* $FreeBSD$ */
> +#include "zstd_kfreebsd.h"
> 
> Added: head/sys/contrib/zstd/lib/freebsd/stdio.h
> ==============================================================================
> --- /dev/null	00:00:00 1970	(empty, because file is newly added)
> +++ head/sys/contrib/zstd/lib/freebsd/stdio.h	Mon Jan  8 20:14:16
> 2018	(r327706) @@ -0,0 +1,3 @@
> +/* This file is in the public domain */
> +/* $FreeBSD$ */
> +#include "zstd_kfreebsd.h"
> 
> Added: head/sys/contrib/zstd/lib/freebsd/stdlib.h
> ==============================================================================
> --- /dev/null	00:00:00 1970	(empty, because file is newly added)
> +++ head/sys/contrib/zstd/lib/freebsd/stdlib.h	Mon Jan  8 20:14:16
> 2018	(r327706) @@ -0,0 +1,3 @@
> +/* This file is in the public domain */
> +/* $FreeBSD$ */
> +#include "zstd_kfreebsd.h"
> 
> Added: head/sys/contrib/zstd/lib/freebsd/string.h
> ==============================================================================
> --- /dev/null	00:00:00 1970	(empty, because file is newly added)
> +++ head/sys/contrib/zstd/lib/freebsd/string.h	Mon Jan  8 20:14:16
> 2018	(r327706) @@ -0,0 +1,3 @@
> +/* This file is in the public domain */
> +/* $FreeBSD$ */
> +#include "zstd_kfreebsd.h"
> 
> Added: head/sys/contrib/zstd/lib/freebsd/zstd_kfreebsd.h
> ==============================================================================
> --- /dev/null	00:00:00 1970	(empty, because file is newly added)
> +++ head/sys/contrib/zstd/lib/freebsd/zstd_kfreebsd.h	Mon Jan  8
> 20:14:16 2018	(r327706) @@ -0,0 +1,56 @@
> +/*-
> + * Copyright (c) 2014-2015 Allan Jude <allanjude at freebsd.org>
> + * All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + *    notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + *    notice, this list of conditions and the following disclaimer in the
> + *    documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> + * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + *
> + * $FreeBSD$
> + */
> +
> +#ifndef ZSTD_KFREEBSD_H
> +#define ZSTD_KFREEBSD_H
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +#ifdef _KERNEL
> +#include <sys/param.h>	/* size_t */
> +#include <sys/systm.h>	/* memcpy, memset */
> +#ifndef BUILDING_ZFS
> +#include <sys/stddef.h>	/* ptrdiff_t */
> +#endif
> +#include <sys/malloc.h>
> +
> +MALLOC_DECLARE(M_ZSTD);
> +
> +#define malloc(x)	(malloc)((x), M_ZSTD, M_WAITOK)
> +#define free(x)		(free)((x), M_ZSTD)
> +/* in zstd's use of calloc, a is always 1 */
> +#define calloc(a,b)	(malloc)((a)*(b), M_ZSTD, M_WAITOK | M_ZERO)
> +#endif
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif /* ZSTD_KFREEBSD_H */
> 
> Added: head/sys/contrib/zstd/lib/freebsd/zstd_kmalloc.c
> ==============================================================================
> --- /dev/null	00:00:00 1970	(empty, because file is newly added)
> +++ head/sys/contrib/zstd/lib/freebsd/zstd_kmalloc.c	Mon Jan  8
> 20:14:16 2018	(r327706) @@ -0,0 +1,33 @@
> +/*-
> + * Copyright (c) 2016-2017 Allan Jude <allanjude at freebsd.org>
> + * All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + *    notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + *    notice, this list of conditions and the following disclaimer in the
> + *    documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> + * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + *
> + * $FreeBSD$
> + */
> +
> +#include <sys/param.h>
> +#include <sys/kernel.h>
> +#include <sys/malloc.h>
> +
> +MALLOC_DEFINE(M_ZSTD, "zstd", "ZSTD Compressor");
> _______________________________________________
> svn-src-head at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/svn-src-head
> To unsubscribe, send any mail to "svn-src-head-unsubscribe at freebsd.org"

It seems that CURRENT is unwilling to build due to the error shown below:

[...]
--- fse_decompress.o ---
In file included from /usr/src/sys/contrib/zstd/lib/common/fse_decompress.c:41:
/usr/src/sys/contrib/zstd/lib/common/bitstream.h:71:12: fatal error:
'immintrin.h' file not found #  include <immintrin.h>   /* support for bextr
(experimental) */ ^~~~~~~~~~~~~
--- modules-all ---