From nobody Tue Apr 04 02:22:49 2023 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PrBPZ2KfWz43hn6 for ; Tue, 4 Apr 2023 02:22:58 +0000 (UTC) (envelope-from tim@timpreston.net) Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PrBPY35nlz41nc for ; Tue, 4 Apr 2023 02:22:57 +0000 (UTC) (envelope-from tim@timpreston.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=timpreston.net header.s=fm2 header.b="L F5NFPD"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=ChWO3lhg; spf=pass (mx1.freebsd.org: domain of tim@timpreston.net designates 64.147.123.21 as permitted sender) smtp.mailfrom=tim@timpreston.net; dmarc=none Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id 6B9CC3200893; Mon, 3 Apr 2023 22:22:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 03 Apr 2023 22:22:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=timpreston.net; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm2; t=1680574975; x= 1680661375; bh=5GFJCWMtScg/S4bqlLLgatv/oOUaAIUYCEiBLaZhqcY=; b=L F5NFPDVnKzaAwSrCSrCCaondG4wakTTMMkkq9uJNaYTUaMrzcBK/3uXhe5XZ90b3 MFxAI8Wo4r7QU5gTS+quas/CFLd1fxvYiAN8AUZ3jyxlGAWZOMav3u3R5eqE4+zJ YSvJBKXlXzHbn0lWQqsi+E0PN4JrM8d7mIiaXIAv1QcknqzlUqAH3M/+PQiNzv8U eZ3N0AsyUq3GBBMmzYAFEwm2US1sb7C9k3U3AnjS2W5v4oi5PHn9+W1/zYzBMDc0 iGQI14Skiriu8SuqLBSZ+2WUVEnpW2ea4JyWjmpZ/SIgwfIRVodhj38NU1qY9CJS 24pOkZCvcQLpnesN40Nvg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1680574975; x=1680661375; bh=5GFJCWMtScg/S 4bqlLLgatv/oOUaAIUYCEiBLaZhqcY=; b=ChWO3lhgsbcOnfhLbkD5H0NyyJkLA osPrXofKrHudviacylLmdhfQ6CI+dmlIlMRJyq4E1iQRiG4G3M9MwG5Uk2V0rhi6 peist8f76rEAnUG/K4ngocTAYhAVcFhuHuLkULOeEQSgZy2Rl1p7wr4CljamV6rJ cZ3W9xZb9Im6sENoIiw/Ij/zNImVGBFQOKqg8Xo1CBoKMFHdkG+y5dN308n80tCG LHddjhHbw7UeLUmZF0MiKP1XQqaj2ydRX38kU6NJVg6IjBf43pylOsMVVZR+3sg7 r5p2kk6iPpt89KxnJr93xm4J3kQScp9ts4BknylnLDJOvBRijHvMX2mZg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeikedggeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurheptgfkffggfgfuvfevfhfhjgesrgdtreertdefjeenucfhrhhomhepvfhimhcu rfhrvghsthhonhcuoehtihhmsehtihhmphhrvghsthhonhdrnhgvtheqnecuggftrfgrth htvghrnhepueffueelffejtdeifffglefhjeekheegveekgffgtefhheehffdutdfftdev hfdunecuffhomhgrihhnpehgihhthhhusgdrtghomhdphhhonhgvhihguhhiuggvrdhnvg htnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhi mhesthhimhhprhgvshhtohhnrdhnvght X-ME-Proxy: Feedback-ID: i1d2040e0:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 3 Apr 2023 22:22:53 -0400 (EDT) Content-Type: multipart/alternative; boundary="------------AZUAt09J05l0S7q1FLmj63aF" Message-ID: <078a1cf8-7ae2-c593-615b-f5f37fa2b3eb@timpreston.net> Date: Tue, 4 Apr 2023 12:22:49 +1000 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Subject: Re: Docker Content-Language: en-AU To: Paul Mather , John Levine Cc: FreeBSD Mailing List , tomek@cedro.info References: <20230329053443.6ADA6B6AFED5@dhcp-8e64.meeting.ietf.org> <8E16D624-2655-4A10-844A-93E4F63E9859@gromit.dlib.vt.edu> From: Tim Preston In-Reply-To: <8E16D624-2655-4A10-844A-93E4F63E9859@gromit.dlib.vt.edu> X-Spamd-Result: default: False [-4.59 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; R_DKIM_ALLOW(-0.20)[timpreston.net:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.21]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.21:from]; RCPT_COUNT_THREE(0.00)[4]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[timpreston.net]; DKIM_TRACE(0.00)[timpreston.net:+,messagingengine.com:+]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PrBPY35nlz41nc X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------AZUAt09J05l0S7q1FLmj63aF Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit It can be done, with a bit of manual tinkering. Here is a gist which explains how to run Docker in a CentOS 8 VM (under bhyve). https://gist.github.com/tehpeh/7e5329d295eca9539e6462f36b6ce9c0 It's a bit out of date but the general idea would be the same for CentOS stream, Alpine etc: install Docker, enable the service, open firewall/networking, nfs mount a local directory. This is pretty much what Docker for Mac does. If you're looking for the Docker hub image repository equivalent for FreeBSD, take a look at Bastille templates or Potluck (https://potluck.honeyguide.net/). However, and this is only my personal opinion, a pre-baked container image repository is a bad idea. Apart from the security issues and recent drama around Docker shutting down free accounts, container images are often set up with default parameters not useful in a production environment (or even your specific dev environment) and are built against a particular kernel version, so may not run as expected on a different kernel version. Again, only my opinion, but you're much better off building your own, private, images targeting the particular OS/Kernel version you use in dev/staging/production. In summary, prefer Dockerfiles over pre-built images. I think the conversation we really need to have is not about copying Docker, but instead how do we consistently create, run, and scale jails across multiple FreeBSD hosts easily. Tim On 2/4/23 02:54, Paul Mather wrote: > On Mar 29, 2023, at 1:34 AM, John Levine wrote: > >> It appears that Tomek CEDRO said: >>> if there are lots of images for linux docker, and docker is linux only >>> solution, there is no reason to talk about it on bsd or even offer some >>> sort of images of bsd for linux right? >> Docker runs on MacOS with a linux emulation layer. FreeBSD already has >> some linux emulation so in principle one could do the same thing, but >> it'd be a lot of work for dubious benefit. > > I disagree it would be of dubious benefit. MacOS is a Tier 1 platform in the Docker ecosystem. Using Docker Desktop on macOS makes using Docker and Kubernetes for development work very easy on that platform, meaning you can stay in the environment you prefer. MacOS is not Linux, but the implementation on there is to use a shim Linux VM via the built-in macOS hypervisor (which, IIRC, is a derivative of bhyve). > > It would be great if the same thing could be done on FreeBSD. It would be beneficial if there was a supported docker machine driver for bhyve on FreeBSD. Right now, I believe the road to running Linux containers on FreeBSD is to use the VirtualBox docker machine driver, which is a bit heavyweight (in terms of added dependencies) for my liking. It would be nice if bhyve could be used to run the shim Linux VM. > > Other than that, much of the tooling to run Docker and Kubernetes is already in ports. But, those (e.g., in the case of Kubernetes) need to point to non-FreeBSD systems that are running the actual containers, pods, etc. It would be nice to be able to do it all on FreeBSD, at least for development and kicking-the-tyres purposes. > > Cheers, > > Paul. > --------------AZUAt09J05l0S7q1FLmj63aF Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit It can be done, with a bit of manual tinkering.

Here is a gist which explains how to run Docker in a CentOS 8 VM (under bhyve).

https://gist.github.com/tehpeh/7e5329d295eca9539e6462f36b6ce9c0

It's a bit out of date but the general idea would be the same for CentOS stream, Alpine etc: install Docker, enable the service, open firewall/networking, nfs mount a local directory. This is pretty much what Docker for Mac does.

If you're looking for the Docker hub image repository equivalent for FreeBSD, take a look at Bastille templates or Potluck (https://potluck.honeyguide.net/).

However, and this is only my personal opinion, a pre-baked container image repository is a bad idea. Apart from the security issues and recent drama around Docker shutting down free accounts, container images are often set up with default parameters not useful in a production environment (or even your specific dev environment) and are built against a particular kernel version, so may not run as expected on a different kernel version.

Again, only my opinion, but you're much better off building your own, private, images targeting the particular OS/Kernel version you use in dev/staging/production. In summary, prefer Dockerfiles over pre-built images.

I think the conversation we really need to have is not about copying Docker, but instead how do we consistently create, run, and scale jails across multiple FreeBSD hosts easily.

Tim


On 2/4/23 02:54, Paul Mather wrote:
On Mar 29, 2023, at 1:34 AM, John Levine <johnl@iecc.com> wrote:


It appears that Tomek CEDRO <tomek@cedro.info> said:

if there are lots of images for linux docker, and docker is linux only
solution, there is no reason to talk about it on bsd or even offer some
sort of images of bsd for linux right?

Docker runs on MacOS with a linux emulation layer.  FreeBSD already has
some linux emulation so in principle one could do the same thing, but
it'd be a lot of work for dubious benefit.


I disagree it would be of dubious benefit.  MacOS is a Tier 1 platform in the Docker ecosystem.  Using Docker Desktop on macOS makes using Docker and Kubernetes for development work very easy on that platform, meaning you can stay in the environment you prefer.  MacOS is not Linux, but the implementation on there is to use a shim Linux VM via the built-in macOS hypervisor (which, IIRC, is a derivative of bhyve).

It would be great if the same thing could be done on FreeBSD.  It would be beneficial if there was a supported docker machine driver for bhyve on FreeBSD.  Right now, I believe the road to running Linux containers on FreeBSD is to use the VirtualBox docker machine driver, which is a bit heavyweight (in terms of added dependencies) for my liking.  It would be nice if bhyve could be used to run the shim Linux VM.

Other than that, much of the tooling to run Docker and Kubernetes is already in ports.  But, those (e.g., in the case of Kubernetes) need to point to non-FreeBSD systems that are running the actual containers, pods, etc.  It would be nice to be able to do it all on FreeBSD, at least for development and kicking-the-tyres purposes.

Cheers,

Paul.

--------------AZUAt09J05l0S7q1FLmj63aF--