![]() ![]() I'm pretty sure this doesn't handle system restart though, so I wouldn't buy it wholesale.ĮxecStartPre=/bin/bash -c 'mkfs -t xfs /dev/nvme1n1'ĮxecStartPre=/bin/bash -c 'mkdir /scratch'ĮxecStartPre=/bin/bash -c 'mount /dev/nvme1n1 /scratch'ĮxecStartPre=/bin/bash -c 'mkdir /scratch/pods'ĮxecStartPre=/bin/bash -c 'mkdir /scratch/docker'ĮxecStartPre=/bin/bash -c 'rm -rf /var/lib/docker'ĮxecStartPre=/bin/bash -c 'ln -s /scratch/pods /var/lib/kubelet/'ĮxecStart=/bin/bash -c 'ln -s /scratch/docker /var/lib/' There's absolutely more work to be done on this, I'd like better conditionality on it so this could just be applied to all nodes without just resulting in a failing systemd unit on differently configured machines, but I think there may be a model here to extend that doesn't require as much finagling around processes that depend on the potential mount points. I have been using this for about a month and a half now and it really just works.ĮxecStartPre=/bin/bash -c 'mkfs.ext4 -L docker_dir /dev/nvme1n1'ĮxecStartPre=/bin/bash -c 'mkdir -p /var/lib/docker/overlay'ĮxecStartPre=/bin/bash -c 'mount LABEL=docker_dir /var/lib/docker/overlay'ĮxecStartPre=/bin/bash -c 'mkfs.ext4 -L empty_dir /dev/nvme2n1'ĮxecStartPre=/bin/bash -c 'mkdir -p /var/lib/kubelet/pods'ĮxecStart=/bin/bash -c 'mount LABEL=empty_dir /var/lib/kubelet/pods' I haven't run into any cases where the RAIDing process has failed. I think there's obviously testing to be done as to how this would consistently operate on various OS's (I didn't need to solve for ubuntu, coreOS, amazon linux, etc), but the process itself was pretty trivial. Outside of unusual install hooks though, if your node is fully ephemeral and the storage is fully ephemeral, you don't need to consider your reboot configuration settings, which was the only other option that there seemed to be. I was actually unable to successfully change this configuration in bootstrap prior to installing mdadm (although I'm sure I was missing something). I did run into issues on the jessie images because of a debconf setting calling for UI interaction for post-install hooks which meant mdadm splashed a blue config screen the first time I tried this manually. Just a note on the complications around software raid that I ran into, it was fairly trivial on the newest k8s stretch AMIs. On startup the instance would examine this configuration and format/mount the disks as specified.Īt this point, though, perhaps instead of actual new configuration options, a simpler solution might just be add some examples to the docs how to use hooks to setup these volumes to your taste, you just have to run a few commands to format and mount the disks at the location of your choice, right? whether or not to format them, what filesystem to use if so, and what path to mount them at, if any. I think ideally instead of having any default / automatic behavior here, adding a configuration section to the instance group that specifies what to do with extra volumes, e.g. ![]() So, I think it would probably be something that should be configurable. It seems like these i3 instances would be great but I wouldn't want kops to format them for me, since ceph would want to take over the device for me.Īlso I've seen mongo recommends xfs as the underlying filesystem rather than ext4. I'm also wondering how to set up ceph in my kops-managed k8s cluster. I think that having this capability is a great idea but significantly complicates the first implementation. ![]() I think that software RAID like is doing should be out of scope for the initial implementation. For /var/lib/docker, we should probably be stopping docker before the mount and starting it again afterwards, especially for systems that enable docker by default (CoreOS). For /var/lib/kubelet, the mount would have to happen before kubelet is started. Using /var/lib/kubelet/pods or /var/lib/docker, however, should be supported. It's my opinion that this should be out of scope for the initial implementation of ephemeral functionality. The use of these ephemeral disks for system directories like /var/lib is super tricky and the implementation varies from system to system, usually involving use of chroot. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |